← You're reading a post on Code for America's Blog Archive. Head back to our main site.

A Step-by-Step Guide to Redeploying Civic Tech in Your Community

One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to “go make a thing.” Build Week is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I’ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

Part One: In Which The Team Has An Idea

Team Indianapolis went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given Mayor Ballard’s plan at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at Open Oakland put together: http://earlyoakland.org/

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

 

Following that link leads to Github which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:

in addition to all of the actual provider data that Early Oakland makes available to parents:

Adopting “the stuff” side of the house is pretty easy:

The data side proved a little more difficult.

Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. ohanapi.org, www.auntbertha.com and openreferral.org are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

We wanted to provide the same ease of access and understandability that Early Oakland offered.

Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called Selenium. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

 

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.

Okay? Okay! Here’s what we did:

Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good number of programing languages. To use it in Ruby all you have to do is install the selenium-webdriver gem:

gem install selenium-webdriver

Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our pull.rb script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to “go make a thing.” Build Week is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I’ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

Part One: In Which The Team Has An Idea

Team Indianapolis went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given Mayor Ballard’s plan at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at Open Oakland put together: http://earlyoakland.org/

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

 

Following that link leads to Github which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:

in addition to all of the actual provider data that Early Oakland makes available to parents:

Adopting “the stuff” side of the house is pretty easy:

The data side proved a little more difficult.

Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. ohanapi.org, www.auntbertha.com and openreferral.org are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

We wanted to provide the same ease of access and understandability that Early Oakland offered.

Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called Selenium. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

 

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.

Okay? Okay! Here’s what we did:

Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good number of programing languages. To use it in Ruby all you have to do is install the selenium-webdriver gem:

gem install selenium-webdriver

Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our pull.rb script:

``

The driver object will have all the methods (in a nice DSL to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:

driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"

If we stop here and run the script here’s what we should see:

Think of all the clicks that saved us!

Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:

```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to “go make a thing.” Build Week is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I’ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

Part One: In Which The Team Has An Idea

Team Indianapolis went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given Mayor Ballard’s plan at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at Open Oakland put together: http://earlyoakland.org/

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

 

Following that link leads to Github which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:

in addition to all of the actual provider data that Early Oakland makes available to parents:

Adopting “the stuff” side of the house is pretty easy:

The data side proved a little more difficult.

Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. ohanapi.org, www.auntbertha.com and openreferral.org are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

We wanted to provide the same ease of access and understandability that Early Oakland offered.

Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called Selenium. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

 

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.

Okay? Okay! Here’s what we did:

Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good number of programing languages. To use it in Ruby all you have to do is install the selenium-webdriver gem:

gem install selenium-webdriver

Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our pull.rb script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to “go make a thing.” Build Week is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I’ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

Part One: In Which The Team Has An Idea

Team Indianapolis went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given Mayor Ballard’s plan at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at Open Oakland put together: http://earlyoakland.org/

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

 

Following that link leads to Github which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:

in addition to all of the actual provider data that Early Oakland makes available to parents:

Adopting “the stuff” side of the house is pretty easy:

The data side proved a little more difficult.

Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. ohanapi.org, www.auntbertha.com and openreferral.org are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

We wanted to provide the same ease of access and understandability that Early Oakland offered.

Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called Selenium. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

 

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.

Okay? Okay! Here’s what we did:

Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good number of programing languages. To use it in Ruby all you have to do is install the selenium-webdriver gem:

gem install selenium-webdriver

Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our pull.rb script:

``

The driver object will have all the methods (in a nice DSL to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:

driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"

If we stop here and run the script here’s what we should see:

Think of all the clicks that saved us!

Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:

  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to “go make a thing.” [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I’ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  

Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:


Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
`````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  

Then we can tell driver to do the actual work of typing in our info with the send_keys method:


#### Bring on the pragmatism

Okay! So the next step seems pretty simple, we need to push that nice “LOGIN to Your Profile button”. Let’s open up our dev tools to see if it has another nice id we can give to our driver:

<img class="alignnone" src="https://lh4.googleusercontent.com/wIN1C9VWUJztU7GNm9t7Aj4_fzzDc5C6h-sNWy1_aZn4EhRSPZ32hI2zKDgNN0PbOvw1GfHFpfjoMvSdwNVNUs32xFdDtSmThuxO-wq6AQqC-ZyhtQEkNvwknb7Lb4oC_uIKPH8" alt="" /> Drat. Luckily for us Selenium also understands the XPath language for figuring out which thing on the page we’re talking about is AND the dev tools will readily give up that information:

<img class="alignnone" src="https://lh5.googleusercontent.com/_jTiaMClptptTnA4nF6wLjMpeeg5h2jO-qsZVkEqYwu5QvQOVs-vKooXuPrOyBA0yUBd2c3h8-_jwVyNZn-IpSpju8s8esRrioNzVcvhI-vaOdn008SX_rgyh7bXUSv7p-jjLsk" alt="" />

Now there are some drawbacks to using XPath in this context. For one, it’s not very human friendly:
  
``````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
`````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  

Bring on the pragmatism

Okay! So the next step seems pretty simple, we need to push that nice “LOGIN to Your Profile button”. Let’s open up our dev tools to see if it has another nice id we can give to our driver:

 Drat. Luckily for us Selenium also understands the XPath language for figuring out which thing on the page we’re talking about is AND the dev tools will readily give up that information:

Now there are some drawbacks to using XPath in this context. For one, it’s not very human friendly:

  
It would be difficult to know exactly what that’s referring to if you didn’t know. Secondly, it’s incredibly brittle. What we’re saying there is “find the first input box in the first table cell of the fourth row of a table in an element with the id ‘login’” if any of those things change our script is hopelessly busted. Again though we hopefully will only need this script once so it should be fine.

Finding the button is pretty quick now, as is telling driver to click on it:
  
```````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
`````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
````` 

#### Bring on the pragmatism

Okay! So the next step seems pretty simple, we need to push that nice “LOGIN to Your Profile button”. Let’s open up our dev tools to see if it has another nice id we can give to our driver:

<img class="alignnone" src="https://lh4.googleusercontent.com/wIN1C9VWUJztU7GNm9t7Aj4_fzzDc5C6h-sNWy1_aZn4EhRSPZ32hI2zKDgNN0PbOvw1GfHFpfjoMvSdwNVNUs32xFdDtSmThuxO-wq6AQqC-ZyhtQEkNvwknb7Lb4oC_uIKPH8" alt="" /> Drat. Luckily for us Selenium also understands the XPath language for figuring out which thing on the page we’re talking about is AND the dev tools will readily give up that information:

<img class="alignnone" src="https://lh5.googleusercontent.com/_jTiaMClptptTnA4nF6wLjMpeeg5h2jO-qsZVkEqYwu5QvQOVs-vKooXuPrOyBA0yUBd2c3h8-_jwVyNZn-IpSpju8s8esRrioNzVcvhI-vaOdn008SX_rgyh7bXUSv7p-jjLsk" alt="" />

Now there are some drawbacks to using XPath in this context. For one, it’s not very human friendly:
  
``````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
`````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
````` 

#### Bring on the pragmatism

Okay! So the next step seems pretty simple, we need to push that nice “LOGIN to Your Profile button”. Let’s open up our dev tools to see if it has another nice id we can give to our driver:

<img class="alignnone" src="https://lh4.googleusercontent.com/wIN1C9VWUJztU7GNm9t7Aj4_fzzDc5C6h-sNWy1_aZn4EhRSPZ32hI2zKDgNN0PbOvw1GfHFpfjoMvSdwNVNUs32xFdDtSmThuxO-wq6AQqC-ZyhtQEkNvwknb7Lb4oC_uIKPH8" alt="" /> Drat. Luckily for us Selenium also understands the XPath language for figuring out which thing on the page we’re talking about is AND the dev tools will readily give up that information:

<img class="alignnone" src="https://lh5.googleusercontent.com/_jTiaMClptptTnA4nF6wLjMpeeg5h2jO-qsZVkEqYwu5QvQOVs-vKooXuPrOyBA0yUBd2c3h8-_jwVyNZn-IpSpju8s8esRrioNzVcvhI-vaOdn008SX_rgyh7bXUSv7p-jjLsk" alt="" />

Now there are some drawbacks to using XPath in this context. For one, it’s not very human friendly:
  

It would be difficult to know exactly what that’s referring to if you didn’t know. Secondly, it’s incredibly brittle. What we’re saying there is “find the first input box in the first table cell of the fourth row of a table in an element with the id ‘login’” if any of those things change our script is hopelessly busted. Again though we hopefully will only need this script once so it should be fine.

Finding the button is pretty quick now, as is telling driver to click on it:

  
We’ll need to combine these techniques to get past the terms and conditions page:
  
````````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
`````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
````` 

#### Bring on the pragmatism

Okay! So the next step seems pretty simple, we need to push that nice “LOGIN to Your Profile button”. Let’s open up our dev tools to see if it has another nice id we can give to our driver:

<img class="alignnone" src="https://lh4.googleusercontent.com/wIN1C9VWUJztU7GNm9t7Aj4_fzzDc5C6h-sNWy1_aZn4EhRSPZ32hI2zKDgNN0PbOvw1GfHFpfjoMvSdwNVNUs32xFdDtSmThuxO-wq6AQqC-ZyhtQEkNvwknb7Lb4oC_uIKPH8" alt="" /> Drat. Luckily for us Selenium also understands the XPath language for figuring out which thing on the page we’re talking about is AND the dev tools will readily give up that information:

<img class="alignnone" src="https://lh5.googleusercontent.com/_jTiaMClptptTnA4nF6wLjMpeeg5h2jO-qsZVkEqYwu5QvQOVs-vKooXuPrOyBA0yUBd2c3h8-_jwVyNZn-IpSpju8s8esRrioNzVcvhI-vaOdn008SX_rgyh7bXUSv7p-jjLsk" alt="" />

Now there are some drawbacks to using XPath in this context. For one, it’s not very human friendly:
  
``````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
`````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
````` 

#### Bring on the pragmatism

Okay! So the next step seems pretty simple, we need to push that nice “LOGIN to Your Profile button”. Let’s open up our dev tools to see if it has another nice id we can give to our driver:

<img class="alignnone" src="https://lh4.googleusercontent.com/wIN1C9VWUJztU7GNm9t7Aj4_fzzDc5C6h-sNWy1_aZn4EhRSPZ32hI2zKDgNN0PbOvw1GfHFpfjoMvSdwNVNUs32xFdDtSmThuxO-wq6AQqC-ZyhtQEkNvwknb7Lb4oC_uIKPH8" alt="" /> Drat. Luckily for us Selenium also understands the XPath language for figuring out which thing on the page we’re talking about is AND the dev tools will readily give up that information:

<img class="alignnone" src="https://lh5.googleusercontent.com/_jTiaMClptptTnA4nF6wLjMpeeg5h2jO-qsZVkEqYwu5QvQOVs-vKooXuPrOyBA0yUBd2c3h8-_jwVyNZn-IpSpju8s8esRrioNzVcvhI-vaOdn008SX_rgyh7bXUSv7p-jjLsk" alt="" />

Now there are some drawbacks to using XPath in this context. For one, it’s not very human friendly:
  
`````` 
  
It would be difficult to know exactly what that’s referring to if you didn’t know. Secondly, it’s incredibly brittle. What we’re saying there is “find the first input box in the first table cell of the fourth row of a table in an element with the id ‘login’” if any of those things change our script is hopelessly busted. Again though we hopefully will only need this script once so it should be fine.

Finding the button is pretty quick now, as is telling driver to click on it:
  
```````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
`````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
```` 

Then we can tell driver to do the actual work of typing in our info with the send_keys method:
  
````` 

#### Bring on the pragmatism

Okay! So the next step seems pretty simple, we need to push that nice “LOGIN to Your Profile button”. Let’s open up our dev tools to see if it has another nice id we can give to our driver:

<img class="alignnone" src="https://lh4.googleusercontent.com/wIN1C9VWUJztU7GNm9t7Aj4_fzzDc5C6h-sNWy1_aZn4EhRSPZ32hI2zKDgNN0PbOvw1GfHFpfjoMvSdwNVNUs32xFdDtSmThuxO-wq6AQqC-ZyhtQEkNvwknb7Lb4oC_uIKPH8" alt="" /> Drat. Luckily for us Selenium also understands the XPath language for figuring out which thing on the page we’re talking about is AND the dev tools will readily give up that information:

<img class="alignnone" src="https://lh5.googleusercontent.com/_jTiaMClptptTnA4nF6wLjMpeeg5h2jO-qsZVkEqYwu5QvQOVs-vKooXuPrOyBA0yUBd2c3h8-_jwVyNZn-IpSpju8s8esRrioNzVcvhI-vaOdn008SX_rgyh7bXUSv7p-jjLsk" alt="" />

Now there are some drawbacks to using XPath in this context. For one, it’s not very human friendly:
  
``````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
``` 
  
Our driver object has a variety of methods to find things like text boxes on the page. Luckily for us, the username and password fields have descriptive ids so we don’t have to do much past telling the driver what those ids are and that they are ids:
  
````One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

``One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n491m8umoXKDaXKZjxb1HLYIdO2Z8Fw-a1H4aEHJsmPLP3zoUwdHdGllU8j9EI_vdQyxc0asqWLSI6bZFc" alt="" />

The data side proved a little more difficult.

#### Service Discovery is Hard

A lot of civic technology efforts have gone toward connecting people to the available services. [ohanapi.org](http://ohanapi.org/), [www.auntbertha.com](https://www.auntbertha.com/) and [openreferral.org](http://openreferral.org/) are just a few of the fantastic projects that are trying different ways to help make those connections. For our purposes, we wanted a data source that contained all the childcare providers in Marion county and also had data on Indiana’s voluntary childcare quality rating and improvement system called Paths to QUALITY™.

Child Care Aware of America publishes a search tool for just such a purpose but it requires registration and can be a bit overwhelming:

<img class="alignnone" src="https://lh6.googleusercontent.com/fK5xdN-KfHQq8CLZnerf45JmzVxxQZ3FavzMaukbpuP3PJIompcMcMP3UGHszn-XWIHCsKwf25o_plSYv6wqCXZ7fBdLycHXiyKDXpEOSwa5-5DElwo-JAJ99cJi_Do1YKqRiLs" alt="" />

We wanted to provide the same ease of access and understandability that Early Oakland offered.

#### Data Integration Can Be Easy

To leverage the data available in the search tool we needed a way to collect what they were publishing. We could have connived our friends and colleagues into registering on the site and writing down all the available information for Marion County, but we figured with less than a week to get what we were after, that route would leave us unpopular and likely missing some data. Instead we decided to automate pulling info directly from the website.

We turned to a toolset normally used to test websites called <a title="Selenium" href="http://www.seleniumhq.org/" target="_blank">Selenium</a>. Selenium lets you write code to simulate a person interacting with a website but can be easily leveraged to pull information as well. Scroll down for part 2 to see how we built that part of the tool.

Once our tool gave us the data we pulled from Child Care Aware, all we had to do was drop it into our modified version of Early Oakland et voilà!

<img class="alignnone" src="https://lh6.googleusercontent.com/UpJjmwWoMwPppQ5iwhObSNCQob1Dd9G0xPGNRZs6sv5NZnn_8CD2CP-zAfRJuawuTaNaWsR_mN2vk4uuCHW9pZbiWeixV47MmHjNusXsnGiZSEOcT9OG9nJq9cs0htNOIEqBPf4" alt="" />

&nbsp;

Since this is a pretty simple informational site that doesn’t need to accept and remember user input or handle lots of updates, it can be made accessible through Github itself. That also means that all the work we did is available for anyone else who might want to do a similar effort in their community or adopt the site to show something completely different from childcare providers.

### Part Two: In Which the Team Gets some Data

Okay! Let’s do some artisanal data pulling! Let’s also get the important disclaimer out of the way: Using Selenium is not the greatest integration method. If the search tool got updated (e.g. a button moves to the other side of the page) Selenium wouldn’t necessarily know how to adapt. This makes our data gathering process incredibly brittle. The good news is if all we’re trying to do is get enough data to validate our ideas about this project, having the process work exactly once is all we need. Once we have that validation we would look to better (but perhaps more development-intensive) methods of collecting information.
  
Okay? Okay! Here’s what we did:

#### Environment setup

Our team is using OSX and already had working development environments with Ruby, RubyGems and Firefox.

#### Installing the Selenium-webdriver gem

Selenium has done a great job of providing access to their tools across a good <a href="http://www.seleniumhq.org/download/#client-drivers" target="_blank">number of programing languages</a>. To use it in Ruby all you have to do is install the selenium-webdriver gem:

`gem install selenium-webdriver`

#### Start scripting

Now that we’ve installed Selenium we can start building our script.

Here are the first few lines we’ll need in our `pull.rb` script:

`` 
  
The driver object will have all the methods (in a nice <a href="http://martinfowler.com/bliki/DslQandA.html" target="_blank">DSL</a> to boot) we need for navigating around the website and finding information on the page. For instance, the first thing we need to do is have Selenium open up the browser and navigate to the tool’s login page:
  
`driver.navigate.to "https://orm.naccrraware.net/orm/ormLogin.action?uid=SOXODVFVF6EL39Q"`
  
If we stop here and run the script here’s what we should see:

<img class="alignnone" src="https://lh4.googleusercontent.com/Y31a3lognqisQ0-oGoI1N-cRVLnuEI22nfVWbxscbSclRo0j1Wjcsdkg_p1IvHQh64BJlC-Gpw3h8MnjRcMEZzBUsn8n_dUwah5t_9Ib84euYtjlvLYgNdvuaE-jj1mczO00p1g" alt="" />
  
Think of all the clicks that saved us!

#### Figure out where stuff is on the page

So the next challenge we face is logging into the site. I went ahead and registered a username and password by hand already, but we’ll need to tell Selenium where on the page to put that information in order to gain access. I didn’t want to hardcode my login info so I’ll pass it in from the command line:
  
```One of our first challenges this year was being turned loose by Code for America in our first few days of knowing one another to &#8220;go make a thing.&#8221; [Build Week](http://www.codeforamerica.org/blog/2015/01/26/doing-a-powerful-force-for-change/) is a new tradition for the fellows. It forces us to figure out how to work together and put our proverbial civic technology money where our civic technology mouths are. I&#8217;ve outlined our step-by-step process, with a few helpful hints for all you civic technologists out there.

### Part One: In Which The Team Has An Idea

[Team Indianapolis](http://www.codeforamerica.org/governments/indianapolis/) went through a condensed hemming and hawing cycle to try and decide what to build, with the most value, in just a week. Any civic technologist worth their salt knows the hawing is always the hard part. Ultimately, we decided on trying to help the parents of Marion county’s approximately 14,000 preschool aged children better understand what early childhood care and education was available to them. This was particularly timely given [Mayor Ballard’s plan](http://www.indianapolisrecorder.com/news/local/article_165ce326-18bc-11e4-a9f8-001a4bcf887a.html) at the time to extend $25 million in early childhood education scholarships and grants to providers over the four years.

In turning out our project we picked up some valuable lessons:

#### Be lazy

We live in a golden age of software openness. That’s particularly true in civic technology. Many, many successful projects freely distribute everything you’d need to bring that project to your community. That means much less work for us and we get to build off what someone has done before us. For our project, we found an awesome project the folks at [Open Oakland](https://www.openoakland.org/) put together: <http://earlyoakland.org/>

[<img class="aligncenter" src="https://lh5.googleusercontent.com/VUXsdEBRpAiEInUq2wtUyt-m4API4SAuj59YLe9guZ9UYs92DP9hVGGu-ELgfWUr-LI_h07wli5GQdfjdsfTiF1gz0atLQ_cqjxB4NmBcK6e3E4IMucbxhk3nErK9w8gg7d2pFY" alt="" />](earlyoakland.org)

Right at the bottom of the site is the best thing a lazy civic technologist could ever hope to see:

<img class="alignleft" src="https://lh4.googleusercontent.com/bfHq8L89uwFPp899sP3jo0qRzTphultzV2zCsTomy5jmvk3THIQLBQrlAubj2npUsLo7SdZoEcwBGZFBoHlKzV_UjklqFEhlt_GRZuJOq4dMCrqQDZdmxcU7y_pAFdWr-8zVakE" alt="" />

&nbsp;

Following that link leads to <a title="Github" href="http://github.com" target="_blank">Github</a> which is a massive collection of all sorts of coding projects that anyone can contribute to or adopt for their own purposes. All the stuff Early Oakland is built from is completely available to us:<img class="alignnone" src="https://lh5.googleusercontent.com/zN5uEF1zwfyBMazHbKqwZEmMNTRgIvcwRh-jgyHV-Osn21foIfRQ2FmJ2P_dLurPdJWM7xC5VhPMZnSIc8qVpc0jip6MmTlg3KA5IpJbAP1kGF7f_wvuQK1vI26wK09nk1Mi6SY" alt="" />

in addition to all of the actual provider data that Early Oakland makes available to parents:

<img class="alignnone" src="https://lh4.googleusercontent.com/IDOoCer9YYgjBlB1WWY1pnjRIAZUbk4XoNQUP68w8oo3LXXoRe2M3ldHRYz-JmlO3NA-X7xQ35I0NqpYAgnUQitblbvCamvEE5rvyOVoloD-_q6W_NKaw10f1D45J-es0S1KMU4" alt="" />

Adopting “the stuff” side of the house is pretty easy:

<img class="alignnone" src="https://lh5.googleusercontent.com/iCY9moRZdogvO-KvZdjmiOe39N9lH0seSF6EvBqKU-cEE7WR6nNf2Bn34D0WqEE9wrfF4xXS60a2rtgQN-3puisz4PvtZAugm-9zOy_SAY7E53sFOhm-sZ8PuBWLszFArWb-AFw" alt="" />

<img class="alignnone" src="https://lh5.googleusercontent.com/9Zyl7HE9uI5uAYpka4yWSLcSR3Ctlq0v2RzKSu8YaSdmpyShns93eui9xfaFAMG5mHc8GRuZoUupKpGS3HGVCS4sUb4YAYmQYcT3GvE_s2GX57fmDM_p0qmeDD6pe0ftkFnuQYs" alt="" />

<img class="alignnone" src="https://lh4.googleusercontent.com/VHP6Fe4IshLdJRU1USgoJ3S-9XV-qugq7iHWVV83BrdxsbI_IYBm_n4