@anonymous-piwik-user opened this Issue on June 12th 2009

I've changed the sendHttpRequest to allow checking for new versions on servers where allow_url_fopen equals zero. My code is tested on my server, and works as expected.

Open core/Piwik.php and replace sendHttpRequest by the following code:

#!php
    static public function sendHttpRequest($url, $timeout)
    {
        // Modified by Uli <m [AT] il [DOT] wolf-u [DOT] li>
        $response = false;
        if (ini_get('allow_url_fopen') == 0) {
            if(function_exists(curl_init)) {
                $ch = <a class='mention' href='https://github.com/curl_init'>@curl_init</a>();
                <a class='mention' href='https://github.com/curl_setopt'>@curl_setopt</a>($ch, CURLOPT_URL, $url);
                <a class='mention' href='https://github.com/curl_setopt'>@curl_setopt</a>($ch, CURLOPT_RETURNTRANSFER,1);
                <a class='mention' href='https://github.com/curl_setopt'>@curl_setopt</a>($ch, CURLOPT_TIMEOUT, $timeout);
                $response = <a class='mention' href='https://github.com/curl_exec'>@curl_exec</a> ($ch);
                <a class='mention' href='https://github.com/curl_close'>@curl_close</a> ($ch);
                unset($ch);
            } else {
                $fsockurl = <a class='mention' href='https://github.com/parse_url'>@parse_url</a>($url);
                if(empty($fsockurl['port'])) { $fsockurl['port'] = 80; }
                $fsock = <a class='mention' href='https://github.com/fsockopen'>@fsockopen</a>($fsockurl['host'],  $fsockurl['port'], $errno, $errstr, $timeout);
                <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "GET " . $fsockurl['path'] . " HTTP/1.0\r\n");
                <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "HOST: " . $fsockurl['host'] . "\r\n");
                <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "Connection: close\r\n\r\n");
                while (!<a class='mention' href='https://github.com/feof'>@feof</a>($fsock)) {
                    if ($get_info) {
                        $response = <a class='mention' href='https://github.com/fread'>@fread</a>($fsock, 1024);
                    } else {
                        if (<a class='mention' href='https://github.com/fgets'>@fgets</a>($fsock, 1024) == "\r\n") {
                            $get_info = true;
                        }
                    }
                }
                <a class='mention' href='https://github.com/fclose'>@fclose</a>($fsock);
                unset($fsockurl, $fsock, $get_info);
            }
        } else {
            // we make sure the request takes less than a few seconds to fail
            // we also set the socket_timeout (for php < 5.2.1)
            $default_socket_timeout = <a class='mention' href='https://github.com/ini_get'>@ini_get</a>('default_socket_timeout');
            <a class='mention' href='https://github.com/ini_set'>@ini_set</a>('default_socket_timeout', $timeout);

            // we create a stream_context (works in php >= 5.2.1)
            $ctx = null;
            if(function_exists('stream_context_create')) {
                    $ctx = stream_context_create(array('http' => array( 'timeout' => $timeout)));
            }
            $response = trim(<a class='mention' href='https://github.com/file_get_contents'>@file_get_contents</a>($url, 0, $ctx));

            // restore the socket_timeout value
            if(!empty($default_socket_timeout))
            {
                    <a class='mention' href='https://github.com/ini_set'>@ini_set</a>('default_socket_timeout', $default_socket_timeout);
            }
        }
        return $response;
    }

Checks by curl if allow_url_fopen = 0. If curl is not available, it gets checked by a socket.

I hope this (or something similar) gets included into one of the next releases ;)

@anonymous-piwik-user commented on June 25th 2009

Attachment: Patch of sendHttpRequest against 0.4.1
patch-sendHttpRequest-0.4.1-1.patch

@robocoder commented on June 13th 2009 Contributor

Why 3 methods? Why not just use sockets if that's the lowest common denominator?

Code comments re: socket version:

  • should call stream_set_timeout()
  • need to concatenate response, otherwise it doesn't work on files > 1K
  • should check the response header for HTTP error code
@anonymous-piwik-user commented on June 13th 2009

Replying to vipsoft:

Why 3 methods? Why not just use sockets if that's the lowest common denominator?
Hmmm good thought. Don't think sockets get disabled by hosters, so yes, this would be the lowest common denominator. You are probably right.

Code comments re: socket version:

  • should call stream_set_timeout()
    I didn't really dig into the socket-thing, i only checked if it works.
  • need to concatenate response, otherwise it doesn't work on files > 1K
    Yeah right, i only saw the updater for this
  • should check the response header for HTTP error code
    Also right, this code is probably too rudimentary

Will update the code, but need to check a few things first.

@anonymous-piwik-user commented on June 13th 2009

I've recoded the sendHttpRequest for sockets with your hints. Furthermore it should now meet the CodingStandard:

static public function sendHttpRequest($url, $timeout)
{
    // Modified by Uli <m [AT] il [DOT] wolf-u [DOT] li>
    $response = false;
    // Parse the url for fsockopen
    $fsockurl = <a class='mention' href='https://github.com/parse_url'>@parse_url</a>($url);
    if(empty($fsockurl['port']))
    {
        $fsockurl['port'] = 80;
    }
    if(!empty($fsockurl['query']))
    {
        $fsockurl['path'].="?" . $fsockurl['query'];
    }
    // Make the request
    $fsock = <a class='mention' href='https://github.com/fsockopen'>@fsockopen</a>($fsockurl['host'],  $fsockurl['port'], $errno, $errstr, $timeout);
    if($fsock||!is_resource($fsock))
    {
        <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "GET " . $fsockurl['path'] . " HTTP/1.0\r\n");
        <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "HOST: " . $fsockurl['host'] . "\r\n");
        <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "Connection: close\r\n\r\n");
        <a class='mention' href='https://github.com/stream_set_blocking'>@stream_set_blocking</a>($fsock, TRUE);
        <a class='mention' href='https://github.com/stream_set_timeout'>@stream_set_timeout</a>($fsock,$timeout);

        while ((!<a class='mention' href='https://github.com/feof'>@feof</a>($fsock)) && (!$streamMetaData['timed_out']))
        {
            // Load data as long as there is data and the connection didn't time out
            $fgetsData .= <a class='mention' href='https://github.com/fgets'>@fgets</a>($fsock, 1024);
            $streamMetaData = <a class='mention' href='https://github.com/stream_get_meta_data'>@stream_get_meta_data</a>($fsock);
            <a class='mention' href='https://github.com/ob_flush'>@ob_flush</a>;
            <a class='mention' href='https://github.com/flush'>@flush</a>();
        }

        <a class='mention' href='https://github.com/fclose'>@fclose</a>($fsock);
        unset($fsockurl, $fsock);
    }

    if (!$streamMetaData['timed_out'] && $fgetsData != "")
    {
        // The connection didn't time out and there is data available
        // Split into headers & body
        $hunks = explode("\r\n\r\n",trim($fgetsData));
        if (is_array($hunks) && count($hunks) >= 2)
        {
            $headers = explode("\n",$hunks[count($hunks) - 2]);
            $body = $hunks[count($hunks) - 1];
            unset($hunks);
            if (is_array($headers) && count($headers) >= 1)
            {
                // Check if the server also told us that everything went well
                switch(trim(strtolower($headers[0])))
                {
                    case 'http/1.0 100 ok':
                    case 'http/1.0 200 ok':
                    case 'http/1.1 100 ok':
                    case 'http/1.1 200 ok':
                        $response = $body;
                    break;
                }
            }
        }
    }
    return $response;
}

Works on my server ;)

@robocoder commented on June 14th 2009 Contributor

Thanks, it looks good (cursory inspection). I'll do a more complete review before commiting.

I'm also going to throw in a couple of additional requirements. (You're welcome to tackle these as well.)

  • Add a check for Content-length (if present in the header).
  • Refactor fetchRemoteFile
@anonymous-piwik-user commented on June 15th 2009

Alright, i've added the check for the parameter "content-length". If it is present, the data gets checked and if the two values are not equal, the output will be false again.

Also removed two possible errors by initializing the variables $fgetsData and $streamMetaData.

/**
 * Sends http request ensuring the request will fail before $timeout seconds
 * Returns the response content (no header, trimmed)
 * <a class='mention' href='https://github.com/author'>@author</a> Uli <m [AT] il [DOT] wolf-u [DOT] li>
 * <a class='mention' href='https://github.com/param'>@param</a> string $url
 * <a class='mention' href='https://github.com/param'>@param</a> int $timeout
 * <a class='mention' href='https://github.com/return'>@return</a> string|false false if request failed
 */
static public function sendHttpRequest($url, $timeout)
{
    $response = false;
    // Parse the url for fsockopen
    $fsockurl = <a class='mention' href='https://github.com/parse_url'>@parse_url</a>($url);
    $fgetsData = "";
    $streamMetaData = array();
    if(empty($fsockurl['port']))
    {
        $fsockurl['port'] = 80;
    }
    if(!empty($fsockurl['query']))
    {
        $fsockurl['path'].="?" . $fsockurl['query'];
    }
    // Make the request
    $fsock = <a class='mention' href='https://github.com/fsockopen'>@fsockopen</a>($fsockurl['host'],  $fsockurl['port'], $errno, $errstr, $timeout);
    if($fsock||!is_resource($fsock))
    {
        <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "GET " . $fsockurl['path'] . " HTTP/1.0\r\n");
        <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "HOST: " . $fsockurl['host'] . "\r\n");
        <a class='mention' href='https://github.com/fputs'>@fputs</a>($fsock, "Connection: close\r\n\r\n");
        <a class='mention' href='https://github.com/stream_set_blocking'>@stream_set_blocking</a>($fsock, TRUE);
        <a class='mention' href='https://github.com/stream_set_timeout'>@stream_set_timeout</a>($fsock,$timeout);

        while ((!<a class='mention' href='https://github.com/feof'>@feof</a>($fsock)) && (!$streamMetaData['timed_out']))
        {
            // Load data as long as there is data and the connection didn't time out
            $fgetsData .= <a class='mention' href='https://github.com/fgets'>@fgets</a>($fsock, 1024);
            $streamMetaData = <a class='mention' href='https://github.com/stream_get_meta_data'>@stream_get_meta_data</a>($fsock);
            <a class='mention' href='https://github.com/ob_flush'>@ob_flush</a>;
            <a class='mention' href='https://github.com/flush'>@flush</a>();
        }

        <a class='mention' href='https://github.com/fclose'>@fclose</a>($fsock);
        unset($fsockurl, $fsock);
    }

    if (!$streamMetaData['timed_out'] && $fgetsData != "")
    {
        // The connection didn't time out and there is data available
        // Split into headers & body
        $hunks = <a class='mention' href='https://github.com/explode'>@explode</a>("\r\n\r\n",$fgetsData);
        if (is_array($hunks) && count($hunks) >= 2)
        {
            $headers = <a class='mention' href='https://github.com/explode'>@explode</a>("\n",$hunks[count($hunks) - 2]);
            $body = $hunks[count($hunks) - 1];
            unset($hunks);
            if (is_array($headers) && count($headers) >= 1)
            {
                // Check if the server also told us that everything went well
                switch(trim(strtolower($headers[0])))
                {
                    case 'http/1.0 100 ok':
                    case 'http/1.0 200 ok':
                    case 'http/1.1 100 ok':
                    case 'http/1.1 200 ok':
                        // Generally the answer will be ok
                        $response = true;
                        // Now check the content-length if available
                        foreach($headers as $header) {
                            if(substr(trim(strtolower($header)),0,15) == "content-length:") {
                                // Check the length against the content
                                if(substr(trim($header),16) != strlen($body)) {
                                    //Reset the response if this is the wrong length
                                    $response = false;
                                }
                            }
                        }
                    break;
                }
            }
        }
    }

    if($response === true) {
        return trim($body);
    } else {
        return false;
    }
}

If i have time in the next few days i will try to refactor fetchRemoteFile, but this will be at the weekend at the earliest.

@robocoder commented on June 15th 2009 Contributor

If you can, please attach your code as a diff. Thanks.

@mattab commented on June 15th 2009 Owner

did you check that it was working when doing the one click upgrade?

@anonymous-piwik-user commented on June 16th 2009

I didn't test sendHttpRequest against the one-click upgrade as the code was tested against various types of content and servers. It does what it should do (as far as i have tested this function).

Perhaps you mean fetchRemoteFile? I didn't have a look into it yet.

@anonymous-piwik-user commented on June 25th 2009

Added a patch against 0.4.1 with one addition, the array $streamMetaData needed to be initialized.

@robocoder commented on August 6th 2009 Contributor

(In [1369]) fixes #793 - rewrite sendHttpRequest() to work when allow_url_fopen=0; also refactor fetchRemoteFile

@robocoder commented on August 12th 2009 Contributor

In [1391], fixes #925, refs #793 - provide curl and stream methods (as originally proposed by Uli)

Order of preference: curl (6s), stream (10s), socket (13s) to d/l latest.zip

Add Piwik::getTransportMethod() and add to Installation system check

This Issue was closed on August 31st 2010
Powered by GitHub Issue Mirror