Scraping links from a listing page into a Google sheet is made simple with this Axiom template. If you don't want to use a Google Sheet, edit the template and tailor it to your requirements using Axiom's no-code bot builder for automating the browser. You could just donwload the links in a CSV.
If you want to scrape links and loop through pages, we recommend you do this in two parts:
- First use an Axiom to scrape the links
- Then another Axiom to scrape pages, see this template
# Install the link scraper bot template
To install this web scraper Axiom click 'Install template'. If you are a new user you will be required to create an Axiom.ai account before you can edit the template.
# Set up a blank Google Sheet to scrape the links into
- Create a new Google Sheet. You can do this in your Chrome browser by entering the shortcut 'sheet.new', presuming you already have an account.
- Name your sheet something like 'Link Scraper’.
- Set up a tab called 'Links'.
# Set up the 'Get data from a webpage' step
- URL - Add the link of the page you wish to scrape links from
- SELECT - Select the links on the page you wish to scrape, set the selector column to link type. Its as easy as point and click, this video shows you how.
- Find pager (optional) - Select the next button if the page has a pager. If it scrolls then do nothing.
- Max Results - When testing set to 2-3 results max, only toggle to all when ready to go live with your Axiom.
# Set up the 'Write Data to a Google Sheet' step
- Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
- Sheet name - Choose the tab you created.
- DATA - Select the Scrape data.
- Clear data before writing | Add to existing data - Set this option to 'Add to existing data'.
# Test run
We always recommed doing a test run - click run then check the output from the scraper. In this case in the sheet.
# Troubleshooting the link scraper
Try the following if your bot is not working. We would also recommend you watch the video to troubleshoot.
- The links come out as text - Set the selector type to link not text.
- Selectors fail and the bot stops, try a new selection.
- The pager is not working - Re-select the next button or try custom selector.
- No output in the sheet - In 'Write Data to a Google Sheet' check data step is connected.