How we got started...

The City Scrapers project is a natural addition to City Bureau's Documenters program, which was launched at City Bureau in April 2015 and, today, pays and trains community members to attend and document local public meetings. In June 2017, City Bureau was awarded a $50,000 grant to prototype a web application that would allow City Bureau's Documenters Manager to more easily manage, coordinate and deploy even more engaged citizens. We began with a question: How can find out when and where every public governance meeting in Chicago is being held? 

Work on the open-source City Scrapers project began in August 2017 in support of this effort. Since then, a community of coders has formed around the public project. Read our blog post below for more on how this community got started, including information on our 2017 Design-a-thon and our partnership with ProPublica Illinois:

What can I learn from working on the City Scrapers project?

A lot about cities! What is Chicago's City Council talking about this week? What are Local School Councils, and what community power do they have? What neighborhood is the police department doing outreach in? Who governs our water?

From building a scraper, you'll gain experience with:

• how the web works (HTTP requests and responses, reading HTML)
• writing functions and tests in Python
• version control and collaborative coding (git and Github)
• a basic data file format (json), working with a schema and data validation
• problem solving, finding patterns, designing robust code

Community Mission

The City Bureau Labs community welcomes contributions from everyone, while prioritizing learning and leadership opportunities for under-represented individuals in tech and journalism.

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion and spiritual beliefs, or sexual identity and orientation.

Ready to code with us?

1. Fill out this form to join our slack channel and meet the community.
2. Read about how we collaborate and review our Code of Conduct.
3. Get started with Installation and Contributing a spider.

We ask all new contributors to start by writing a spider and its documentation or fixing a bug in an existing one in order to gain familiarity with our code and culture. Reach out on slack for support if you need it. We also meet up in person to work together regularly, and post about upcoming meetups in slack.

For those familiar with the project, please see the help-wanted Github issues.

Don't want to code?

Join our slack channel (chatroom) to discuss ideas and meet the community!

We have ongoing conversations about what sort of data we should collect and how it should be collected. Help us make these decisions by commenting on issues with a non-coding label.

Good with Google? Help us research sources for public meetings. Answer questions like: Are we scraping events from the right websites? Are there local agencies that we're missing? Should events be updated manually or by a scraper? You can dive in today by triaging event sources on these issues.

Join us in-person at our Open Coding Sessions!

We meet every other Monday to code together, tackle issues and engage in some collective decision-making. The Open Coding Sessions are open to coders and non-coders alike. Be sure to bring a computer to each session so we can get you started.

Become a Documenter

The City Scrapers project operates as the data collection system for City Bureau's Documenters program, which pays and trains community members to document public meetings and engage in the production of journalism, news and information. You can become a Documenter by applying here (everyone is accepted) and attending at least one City Bureau sponsored training.