Python makes it easy to monitor your sites!

One of my friends has a web development shop. As a part of their offerings, they offer custom website development at all price points and hosted on all kinds of servers- ranging from shared hosting providers, to VPS’s to major cloud services such as AWS or Azure. Some of the sites are doing pretty well in terms of traffic.

One of the major concerns, especially for low cost hosts is that they tend to have poor reliability and need supervision to ensure uptimes. My friend’s organization was using a commercial service to monitor these sites and the ever expanding portfolio of sites to maintain was causing the costs to creep up continuously.

I learnt of this predicament in a casual phone call and was wondering if I could hack together a quick and dirty solution to get the job done in a couple of hours at most.

The solution was to just get at the http status codes using the fantastic Requests module in Python. It was my first time using it, and I found it pretty easy going to get it up and running.

import datetime
import time
import requests
def testsite(urlstring):
    try:
        r = requests.head(urlstring)
        print(r.status_code)
        if r.status_code != 200:
            requests.get("<<API ENDPOINT>>?url=" + urlstring + " error:" +str(r.status_code))
    except requests.ConnectionError:
        print("failed to connect")
        requests.get(<<API ENDPOINT>>?url=" + urlstring + " error: Failed to connect")
if __name__ == '__main__':
    
    while True:
        testsite('http://google.com/') #your site here
        testsite('https://www.stackoverflow.in/') #add yours
        time.sleep(300)

Let us go through the testsite(urlstring) part which is the meat of the logic.

The

r = requests.head(urlstring)

retrieves the “headers only” from a URL, which is pretty speedy(since the content is not fetched). This returns the requests.response object, which contains the status code as a parameter. My friend quickly threw together a simple API endpoint inside their existing server application to log the script result. All it does is to trigger an e-mail to the server administrator with contents of the script, so that the server administrator instantly knows which site is down. Once the data is on the server, the reporting and other analyses possibilities open up.

if r.status_code != 200:
    requests.get("<<API ENDPOINT>>?url=" + urlstring + " error:" +str(r.status_code))

Additionally, if the server itself goes kaput, the retrieval throws an exception and we catch that as yet another error.

except requests.ConnectionError:
        print("failed to connect")
        requests.get(<<API ENDPOINT>>?url=" + urlstring + " error: Failed to connect")

If you do not have access to an API endpoint, you can log the same to a file, like so:

f= open("log.txt", "a+")
f.write(urlstring + ", " + str(r.status_code) + ", " + str(datetime.datetime.now()) + "\r\n" )
f.close()

The main function just checks the listed sites for ever every 5 minutes (300 seconds). We set this up on a Raspberry Pi connected to the office LAN and this is happily chugging along for a few days now.

Test Log for illustration

There was a further modification to detect if the site was hacked and the page replaced with gibberish, which we detected using some clever Selenium based python code, but that is a story for another day.