How to automatically ping search engine crawlers using python.
In today’s post we will see how
to make a python bot that will automatically ping the search engine crawlers to
come and crawl your website/blog. For this. we will be using the help of
ping-o-matic.com.
Ping-o-matic is a website that
dose the same thing i.e it pings the search engine crawlers to come and crawl
your website/blog. This helps in getting better ranks in the search results. I
don’t know if it really works, but it is definitely worth a try.
Once you go into the ping-o-matic
website, it will ask you for your website name, website URL, etc. once you hit
submit, it will start the pinging. After that it will ask you to bookmark the
page so that you don’t have to type in the details again and again. The next
time you want to ping, you just have to click on the bookmark.
What we will be doing is, we will
be opening the bookmarked URL through python. It is not at all complicated. The
coding is very simple. Because most of the work will done by ping-o-matic.
If you have never worked with the
web in python, you don’t have to worry this does not require any experience.
For opening any URL through python, there are many ways. One is the raw sockets
way, which is a lower level of doing it, but it is not necessary. Two there is
module called ‘requests’, which allows you to send GET, POST and PUT requests
to any web server. Three, is using urllib module for sending a simple GET
request.
The syntax for accessing the URL
or sending a GET request is very simple. First you have to import the urllib2
module. Then you have to call the function urlopen() with the bookmarked URL as
the argument.
That’s about it. That will be as
good as clicking the bookmark. But this will only ping once. If we want it to
keep sending pings forever, then we will have to run it in an infinite loop.
But of course there is a minimum amount of waiting time before you can ping the
second time. So we will have to call the urlopen() function after a certain
time delay. Just like C/C++ has the delay() function, python has something
similar. It has the sleep() function, which comes under the module ‘time’. But
in delay() the time you specify as argument is in milliseconds. Whereas in
sleep() it is in seconds. Refer to the below picture to see how to use the
sleep() function in an infinite loop.
So that is the whole code
required. As you can see, there is a time delay of 1800 seconds i.e. 30 min. So the program will open the
bookmarked URL every 30 min. you can even increase the frequency of pinging but
I don’t think you can go below 10-15 min.
I have added an exception for URL
error i.e. if the URL is wrong, of if the site is not responding or any such
issue, it will not crash the program and print ‘URL error’ instead.
Now for running this bot, you can
either execute it and minimize it in your computer. But you will have to keep
it on forever. Or you can host the script on a cloud service like Heroku which
will keep it running for you.
I personally use this for this
blog, and the results are astonishing. So I recommend that you also try it
yourself. I will cover how to use Heroku in another post. Till then you can
refer to Youtube for clarity. So that’s it for today’s post, will be back with
more useful ideas and concepts. Until then, happy coding.