In this post I will give a brief overview over how to build a scraper for google maps. I will share little parts of code but it really depends on how you plan to scrape them and what language you will be using. I will mostly discuss theory and you will need to find the specifics your self once you pick a programing language. With that said let\u2019s get starting talking about making a google maps lead extractor. <\/p>\n\n\n\n
First off, we need to pick a language for our scraper. What\nyou pick is entirely up to you and how you plan to use your scraper. Local\nScraper is build using C# because that was the easiest way for me to make a\ndesktop application. Python is super common for scrapers and I use it for many\nprojects! But when it comes to making a UI and packaging for desktop use it\ngets complex. Using C# allowed me to make my UI in Visual Studio and saved me a\nton of headaches. With python you will most likely be running your scraper from\na console\/terminal, if you chose JavaScript it would be similar. I don\u2019t think\nany one language is better than the other so just pick what works for you.<\/p>\n\n\n\n
The main thing you will need is a browser remote control system such as Selenium <\/a>or Puppeteer<\/a>. These are systems that will allow us to take control of a copy of Google Chrome and allows us to send commands to the browser. Since google maps only loads in a browser you have to use a browser to view them. Personally, I chose Selenium since I am most familiar with them since they have been around the longest. They also support the most languages because of this. It was super easy to open NuGet in Visual Studio and grab Selenium for C#. I have also used Selenium with python as well and its just as easy to install. Puppeteer is the newer system and is based on NodeJS but some have ported the package over to other languages as well Puppeteer Sharp is the .Net port for C#. <\/p>\n\n\n\n Now that we have that done, we want our google maps extractor<\/a> to open a copy of our browser and browse directly to Google Maps. This step is very straight forward and is covered in all documentation. After that we want to be able to click on each listing and also go though all pages of results until they don\u2019t give you any more pages. <\/p>\n\n\n\n To figure out the next page button we will want to inspect\nthe element in the chrome dev tools and find a unique identifier that we can\ntest for. In this case I found that the button\u2019s \u201cjsaction\u201d will contain \u201cpane.paginationSection.nextPage\u201d\nso we can test for that using xpath. That looks something like this \u201c.\/\/button[contains(@jsaction,\\”pane.paginationSection.nextPage\\”)]\u201d\nSo to make our google maps extractor click on the next page we can test for\nthis xpath. If it exists then we know there is a next page. We can also tell our\ncontrol system to click on this xpath. Pretty simple, until we get to the final\npage of results. This where they trick the scraper. On the last page this next page\nbutton will still exist in the source code! It will always test true and our\nscraper will get stuck. The difference is that on the last page the button will\nadd \u201cdisabled=\u2019true\u2019\u201d to its code. We can use that to make a new test. Which\nlooks something like this “.\/\/button[contains(@jsaction,\\”pane.paginationSection.nextPage\\”)\nand @disabled=\\”true\\”]” Our new xpath is now testing for the\nbutton AND testing if it is disabled or not. If it is disabled then we know we\nare on the last page or there simply is no next page. <\/p>\n\n\n\n Our google maps extractor\ncan now go though all the pages but how can it click on each listing? Again, we\nneed to find a unique identifier by going over the source code in the inspector.\nTry to see what each listing has in common that you can click on, check on the child\nelements like the business title but also check the div that each listing is\nin. In our case what I found was \u201cclass=\u2019a4gq8e-aVTXAb-haAclf-jRmmHf-hSRGPd\u2019\u201d\nthis gibber\u2019ish used to be normal text but google for some reason has\nobfuscated it to make it harder to read. Each listing has this class so we can\nuse that to click on each one. The main issue is that we have this class but\nthere is nothing else unique between the listings. This means we need to make a\nway to click on the 2nd<\/sup> and 3rd<\/sup> item ourselves. <\/p>\n\n\n\n What I ended up doing is\ndoing a count of how many times we found that class in the search results source\ncode. I saved this to a variable so I knew exactly how many listings were on\nthe page, because this changes with each search. Its sometimes 2 listings, its\nsometimes 5 listings, maybe it has 5 pages of full results but the last page\nonly has 2 listings. Each time you load a new page of results you need to rerun\nthis counting system. Clicked next page? Wait a few seconds and then run the\nlisting counter! <\/p>\n\n\n\n Now that we now how many\nresults are on the page we can make a loop to click on each listing. To do so\nwe are going to make a loop that uses are counter variable, say its 10. We need\nto loop ten times. To click on the items we need to click on the class we found\nbut at the loop position. We can use loop position 1 to click on the second\nlisting, position 2 to click on the third one, and so on. To do this I used Selenium\u2019s\nability to run javascript in the browser to activate a click. This code looks something\nlike this “document.getElementsByClassName(\\”a4gq8e-aVTXAb-haAclf-jRmmHf-hSRGPd\\”)[“+counter+”].click()”\nyou can see that I have added [counter] onto the class so we can tell the lead\nextractor which one to click. <\/p>\n\n\n\n After we click on the listing we then want to grab the source\ncode of the listing page, we can then parse this over. To get back to the\nlisting page we click the back button or we have the browser run the back\ncommand. Then we can do the same for the rest of the listings. In the end we\nhave a google maps extractor that will open the browser, go through the pages,\nand click on each listing one by one. Of course the final product will need to\nbe a bit more complex but this is the basic framework of what you want the extractor\ndo it. <\/p>\n\n\n\n If your business is in need of data extracted from google maps and you don\u2019t want to create your own program from scratch Local Scraper can help! We have already done the work and already built the scraper, used by thousands of companies for over 8 years you know you can trust us to not disappear overnight. Our google maps extractor<\/a> gathers as much data as we can and is always kept up to date. If anything changes at google maps, and it will! Know that we will be on the case to fix the issue and keep the program up and running. Head on over to the sales page if you want to know more and if not good luck in your scraper building and I hope this overview has helped your project. <\/p>\n\n\n\n