I learned to scrape websites after I brought Backstage’s customer care and print fulfillment in house.
I was Head of Customer Care and Community. A few VLOOKUPs and a biweekly mail file helped four of us, me plus three other in-house reps, replace a 17-person print-fulfillment and customer call center. We oversaw the paying subscriber base growing from 30,000 to 60,000.
Customers could cancel by phone or email. The work was manual either way.
That is still the point of automation for me: give people more time for the parts of life nothing can automate.
The free year
Before I worked there, I had volunteered at a Backstage event in exchange for a free subscription. That is the honest reason I showed up.
Years later, I wondered whether we could give the same break to actors who were just starting out.
Showcase pages tell the industry how these new professionals want to be reached. They had published those pages because they wanted to be discovered. I needed to collect the information from those pages into one usable list.
I took the idea to our CTO and started learning Ruby. In The Bastards Book of Ruby, I found Nokogiri. It reads HTML or XML and returns the elements you ask for. I used it to turn scores of public showcase pages into a clean list of actors.
The scraper let me offer those actors a free year. Hundreds accepted, and 90% of them went on to become paying subscribers.
I did not want retention to be my career anymore. I wanted acquisition.
The morning list
At my next job, Argyle Executive Forum, another scraper built sales a new call list before breakfast.
The sales development reps there sold side events around conferences run by Gartner and The Conference Board. A strong prospect was a company that had just sponsored a competitor’s event.
To find those companies, reps opened competitor sites every morning, scanned walls of logos, tried to spot what had changed, identified the company behind each new logo, and hunted for the right field marketer. Then they did it again the next day.
It was a good sales play buried under tedious research.
I scraped the public pages, used reverse image search to identify the logos, checked each company against Salesforce, found the right contact, and put a fresh call list in a spreadsheet.
The tools available now make this look simple. In 2015, it was not. The system gave sellers back hours every week as Argyle grew, acquired CFO.com, and expanded internationally.
What I kept
At Backstage, public showcase pages helped me find actors who later subscribed. At Argyle, competitor pages gave sellers a call list. Both projects got useful information to the people who could act on it, before they spent hours gathering it themselves.
I still use the rule: your data pipeline predicts your revenue pipeline.
When the inputs are stale or incomplete, the team researches instead of selling. More people can absorb the extra work for a while. The bad data is still there.
That is why I care about AI answer engines.
AI answer engines do the same basic job at enormous scale. They read web pages, break the information into usable pieces, and answer a buyer’s question without necessarily sending the buyer to the source. Whether they name your company depends on what they can find, understand, and trust.
I built a meter that asks the engines real buying questions and records which brands appear. The job felt familiar: collect what is public, clean it up, and turn it into a list someone can use. This time, the list tells a marketer what the machines noticed. You can run it yourself, or read how the answer layer works.
I did not set out to make public data a career. I kept finding work nobody should have been doing by hand. Web scraping changed my life because it gave me a way to solve that problem: collect what is public, clean it up, and give it to someone who can act on it. I have been doing versions of that ever since.