What are the names of all the people wearing purple shirts in Disney World’s Magic Kingdom right now? Don’t worry, I understand how ridiculous that question sounds. Even for anyone in the Magic Kingdom, the task is enormous. You would have to canvas the entire park for anyone wearing a purple shirt. Then, for each one, you have to ask their name and document it. And no matter how fast you get at it, by the time you are done, there is surely a new influx of purple-shirt-wearing Disney fans walking through the gates.
I bring up this as an analogy to web data. There is such an incomprehensible amount of it, finding the information you need is a mammoth task. Even trying to be really specific leaves you with an impossible job. In the example above, I mention Magic Kingdom only. Even specifying that park apart from the others is still ridiculous. And like manually searching for those people, manually searching for that data is long, hard work with little reward. You can spend hours looking up home prices in Mobile, Alabama and still have hardly any idea what the average is. However, there is help. You can use scraping tools to turn the tide.
Pulling Data Automatically
Let’s go back to the Magic Kingdom. You have the same task, but this time, you have a tool. You can access the metal detector occupant database. This imaginary database stores a few pieces of data. Two of which are the name of the person and the color of the shirt. Now, all you have to do is download that data and filter by shirt color. There’s the entire list in a tiny fraction of the time.
This same idea works with web scraping. With the right tool, you do not have to manually search for the information you need. You input the details you need, and the tool will automatically pull the relevant data and deliver it to you. You can scrape information from entire websites based on your parameters. When the tool finishes its job, you are presented with the information that you can filter and sort as you need.
The Proxy’s Role
Unfortunately, there is a little more to it than that. Websites do not like having their data scraped. This can be for a few different reasons, but one has to do with security. Over the years, a popular form of digital warfare was distributed denial of service (DDoS) attacks. Unleashing DDoS on a website meant bombarding it with countless requests all at the same time. The website would do its best to parse the thousands upon thousands of requests, slow down to a crawl, and then lock up. These attacks have cost businesses a lot of money. So now, they do their best to protect against them.
Why bring this up? Well, there is another act that sends thousands of requests all at the same time: scraping. While smart scraping will not put more strain on the server than it can handle, webmasters like to play things safe. They have bots that are programmed to notice when a single source sends requests at an inhuman rate. The first line of defence is to block that source. If this happens, your quest for data grinds to a halt.
So how can proxies help? Well, using proxies from reliable proxy providers spreads out those requests. It works in tandem with your scraping tool. Instead of sending all of your requests directly from you, there are a handful of proxies working together to make it look like the requests are coming from all over. Each proxy moves closer to human speed, but their cumulative speed is still incredible. And the tricks do not stop there.
What if your scraping target still gets wise and starts banning your proxies? Well, you should have access to a lot of proxies, so you can just swap them out. Your proxies can even be configured with your tool to be cycled out automatically. If a proxy’s connection is denied, it gets replaced with a fresh one. You do not need to worry about digging into the piggy bank for this. Buying proxies in bulk is surprisingly inexpensive. As technology has grown, the costs of proxies have gotten more and more affordable.
Conclusion
It is true, the Magic Kingdom example is not realistic. The idea of needing to know such specific information about random visitors to a theme park is ludicrous. And surely there is not any software with the metal detectors that can store our shirt colors and names. I say sure, because, unfortunately, I cannot be 100% sure about that. But as crazy as it is, I made that example for a reason.
The worldwide web is unlike anything we have experienced before or since. Information exists for incredibly mundane topics and details. At first look, they have no use for anyone. Why would it matter when I took the photo I posted in a travel message board? For you, it likely doesn’t. However, if you wanted to know who to market a new telephoto lens too, you might be interested to know if that photo was taken recently or not. In our Magic Kingdom example, maybe you personalize shirts at a park kiosk. You just got in a stock of purple shirts, and you want to know which names you want to start with. While the information you could get would not guarantee sales, it would give you a much better start than guessing.
When everything is said and done, it is difficult to find a more realistic comparison. Methods of data collection outside of a computer are not at the same level. We are lucky that we have access to such a huge database of information. We are far luckier that we have tools and proxies to help us dig through all of that data for the bits we need.
