Jelly and Custard

Determining if a URL exists with Curl

May 31st, 2006

Its quite common for people to enter in their URL when signing up - but what if you want to verify that this is a real page? You can validate the URL using a regular expression up to a point, but all that tells us is that the URL is well formed. What I wanted to do was to check that the page exists - i.e. that we don't get a 404 for it.

Luckily, this is quite easy if you have the Curl extension installed.

Using Curl in PHP

Curl is a collection of client URL library functions that we can use to perform a multitude of URL related activities. You can send a POST or file upload to a URL, use it as a proxy, send/receive XML requests, and much more. In fact, there are over 100 configuration options that can customise this to your needs.

Still, for this tutorial, all we want is to check that a URL exists. We can do this by using curl to grab the headers when requesting the URL and check the Status: header returned.

The result from running the above code is:

HTTP/1.1 200 OK
Date: Wed, 31 May 2006 12:25:26 GMT
Server: Apache
X-Powered-By: PHP/5.1.2
X-Pingback: http://www.jellyandcustard.com/xmlrpc.php
Status: 200 OK
Content-Type: text/html; charset=UTF-8

As you can see, we have our Status: 200 OK HTTP Code which means the page exists. If we get a 404 HTTP code then the page doesn't exist. A list of HTTP Codes can be found at Wikipedia.

If the domain can't be resolved, then $data will return false, which we can check against with a simple if() statement.

In order to utilise the information returned, we need to use a regular expression to grab the HTTP code, and then based upon that, we can let the user know our results:

Array
(
    [0] => Status: 200
    [1] => 200
)

So $matches[1] has our error code. If this matches 200 then the page exists, and if it is 404, then our page doesn't exist.

A slight change has to be made for pages that use a 301/302/307 redirect. We need to instruct Curl to follow the redirect, and report what's found. These redirects produce a lot of headers, as seen when testing http://ukdomains.jellyandcustard.com:

HTTP/1.1 302 Found
Date: Wed, 31 May 2006 12:50:38 GMT
Server: Apache
Location: http://www.123-reg.co.uk/affiliate.cgi?id=AF106554
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 302 Found
Date: Wed, 31 May 2006 12:50:42 GMT
Server: Apache/2.0.53
URL: http://www.123-reg.co.uk/
Set-Cookie: 123reg_affiliate=106554-92bab777b0ae96c938107985db7a98e4; path=/secure/; expires=Sat, 28-May-2016 12:50:43 GMT; secure
Location: http://www.123-reg.co.uk/
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 200 OK
Date: Wed, 31 May 2006 12:50:43 GMT
Server: Apache/2.0.53
Accept-Ranges: bytes
Content-Length: 33854
Content-Type: text/html

Array
(
    [0] => Array
        (
            [0] => HTTP/1.1 302
            [1] => HTTP/1.1 302
            [2] => HTTP/1.1 200
        )

    [1] => Array
        (
            [0] => 302
            [1] => 302
            [2] => 200
        )

)

Our Status code can be found by:

Which we can now check for 200 or 404 etc to make sure that you've been given a real URL, and not one that is made up. So the entire code all together:

Enjoy!

 Add to del.icio.us    Digg this    Technorati

Related Posts:

• PHP, cURL, HTTP PUT, SSL and Basic Authentication
• PHP 5.1.3 Released
• Odd and Even Numbers
• File Uploads
• MySQL Events

Entry Filed under: PHP