Random idea -> Data to Images

[SplitIce]

Already posted go back a couple pages.

Ive tested on PHP6

<?
    $im = new ImageContainer('file.txt','image');
    $im->CreateSplit(1024*50);//50kb split.
    /*
    Note: this is filesized splits not the output image size, the output size will depend on compression, if its text you can expect a output size of roughly 20% of the file
    else if its actually binary you can expect around 20% overhead.
    */

    $im2 = new ImageContainerExtract('image0.png');
    $im2->Extract_File('php://output');
    error_reporting(E_ALL);

    class ImageContainer {
        private $file = '';
        private $number_total = 0;
        private $out = '';

        function ImageContainer($file,$out){
            $this->file = $file;
            if(!file_exists($file)){
                throw new Exception('File does not exist');
            }
            $this->out = $out;
        }

        function CreateSplit($split){
            $fs = filesize($this->file);
            if($split>$fs){
                $split = $fs;
            }
            $this->number_total = ceil($fs/$split);//Total number of splits
            $fp = fopen($this->file,'rb');
            $i = 0;
            while($fs>0){
                $h = $w = floor(sqrt($split));
                $h += ceil((sqrt($split)-$w)/$w);
                $fs -= $h*$w;
                $this->file2image($i,$fp,$this->out.$i.'.png',$w,$h,$split);
                $i++;
            }
            fclose($fp);
        }

        function file2image($split_no,$fp,$image,$w,$h,$split){
            $header = '[I]'.dechex($split_no).'.'.dechex($this->number_total).'.'.base64_encode(basename($this->file)).'|';
            if(strlen($header)>$w*$h-$split){
                $h++;
            }

            $gd = imagecreate($w,$h);

            $color = array();
            for($i=0;$i<=255;$i++){
                $color[$i] = imagecolorallocate($gd,$i,$i,$i);
            }
            $wc = $hc =0;        

            for($hc=0;$hc<$h;$hc++){
                for($wc=0;$wc<$w;$wc++){
                    $r = '';
                    if($header) {
                        $r=$header{0};
                        $header = substr($header,1);
                    }else $r = fread($fp,1);
                    imagesetpixel($gd,$wc,$hc,$color[ord($r)]);
                }
            }
            imagepng($gd,$image,9,PNG_NO_FILTER);

            imagedestroy($gd);
        }
    }

    class ImageContainerExtract {
        private $file = '';
        private $efile = '';
        private $data = array();
        private $parts = 1;

        function ImageContainerExtract($file){
            $this->file = $file;
            $this->data[0] = $this->decode_image_raw($file);
            $part_number = $this->ReadHeader(0,true);
            if($part_number!==0){
                throw new Exception('Not first part.');
            }
            for($i=1;$i<$this->parts;$i++){
                $filen = str_replace('0.png',$i.'.png',$file);
                $this->data[$i] = $this->decode_image_raw($filen);
            }
        }

        function ReadHeader($index,$strip=true){
            $mark = substr($this->data[$index],0,3);
            switch($mark){
                case '[I]':
                    $endof = strpos($this->data[$index],'|');
                    $header = substr($this->data[$index],0,$endof);
                    if($strip){
                        $this->data[$index] = substr($this->data[$index],$endof+1);
                    }
                    list($part_number,$this->parts,$this->efile) = explode('.',$header);
                    $part_number = hexdec($part_number);
                    $this->parts = hexdec($this->parts);
                    $this->efile = base64_decode($this->efile);
                    return $part_number;
                    break;
            }
        }

        function decode_image_raw($image){
            $ret = '';
            $gd = imagecreatefrompng($image);
            list($w,$h) = getimagesize($image);
            for($ih=0;$ih<$h;$ih++){
                for($iw=0;$iw<$w;$iw++){
                    $rgb = imagecolorat($gd,$iw,$ih);
                    $ret .= chr($rgb);
                }
            }
            imagedestroy($gd);
            return $ret;
        }

        function Extract_File(){
            $fp = fopen($this->efile,'wb');
            foreach($this->data as $d){
                fwrite($fp,$d);
            }
            fclose($fp);
        }
    }
?>

[Hyperz]

Time to bench it. I forgot how to measure time difference in php though.

Something a la:

let l = (float (new FileInfo(f)).Length) / float 1024
let mb = (float (new FileInfo(f)).Length) / float 1024 / float 1024
let s = DateTime.Now

CreateImages f t
|> ignore

let e = DateTime.Now

printfn "Operation took %s seconds @ %s KB/s (%s MB)!" (e.Subtract(s).TotalSeconds.ToString("n")) ((l / e.Subtract(s).TotalSeconds).ToString("n")) (mb.ToString("n"))

Help ^^

[SplitIce]

Windows clock is how I did it.

Also btw disable compression (change imagepng(...,...,9) to imagepng(...,...,0)) as I did for my test.

Howeaver microtime() or time() will do it.

Just a warning, it will depend greatly on the amount of ram and alot of other independant variables.

Also larger files are faster per mb, due to the memory allocation overheads in php.

[Hyperz]

Mkay. I forgot to change the file size because I read your reply after first trying to test it. But the difference in results is big enough to say that upping the file size would have made no real difference. If you want I can send you the exe to test yourself if you want.

Spec: Phenom II x4 955BE @ stock (3.2ghz) / 4GB RAM DDR2 800
PHP: 5.2.9
.NET: 4 Beta 2 using F# Beta

Results:


May that case be settled

[SplitIce]

lol well I used a file generated from str_repeat = exactly 50mb

Using time() and no compression.
5 seconds

Using time() and compression
8 seconds

script (compression):

<?
ini_set('memory_limit','2G');
set_time_limit(0);
file_put_contents('file.txt',str_repeat('-',1024*1024*50));
$start = time();
    $im = new ImageContainer('file.txt','image');
    $im->CreateSplit(1024*1024*1024);//1gb split.
    /*
    Note: this is filesized splits not the output image size, the output size will depend on compression, if its text you can expect a output size of roughly 20% of the file
    else if its actually binary you can expect around 20% overhead.
    */
die('Time (sec): '.(time()-$start));
    $im2 = new ImageContainerExtract('image0.png');
    $im2->Extract_File('php://output');
    error_reporting(E_ALL);

    class ImageContainer {
        private $file = '';
        private $number_total = 0;
        private $out = '';

        function ImageContainer($file,$out){
            $this->file = $file;
            if(!file_exists($file)){
                throw new Exception('File does not exist');
            }
            $this->out = $out;
        }

        function CreateSplit($split){
            $fs = filesize($this->file);
            if($split>$fs){
                $split = $fs;
            }
            $this->number_total = ceil($fs/$split);//Total number of splits
            $fp = fopen($this->file,'rb');
            $i = 0;
            while($fs>0){
                $h = $w = floor(sqrt($split));
                $h += ceil((sqrt($split)-$w)/$w);
                $fs -= $h*$w;
                $this->file2image($i,$fp,$this->out.$i.'.png',$w,$h,$split);
                $i++;
            }
            fclose($fp);
        }

        function file2image($split_no,$fp,$image,$w,$h,$split){
            $header = '[I]'.dechex($split_no).'.'.dechex($this->number_total).'.'.base64_encode(basename($this->file)).'|';
            if(strlen($header)>$w*$h-$split){
                $h++;
            }

            $gd = imagecreate($w,$h);

            $color = array();
            for($i=0;$i<=255;$i++){
                $color[$i] = imagecolorallocate($gd,$i,$i,$i);
            }
            $wc = $hc =0;        

            for($hc=0;$hc<$h;$hc++){
                for($wc=0;$wc<$w;$wc++){
                    $r = '';
                    if($header) {
                        $r=$header{0};
                        $header = substr($header,1);
                    }else $r = fread($fp,1);
                    imagesetpixel($gd,$wc,$hc,$color[ord($r)]);
                }
            }
            imagepng($gd,$image,9,PNG_NO_FILTER);

            imagedestroy($gd);
        }
    }

    class ImageContainerExtract {
        private $file = '';
        private $efile = '';
        private $data = array();
        private $parts = 1;

        function ImageContainerExtract($file){
            $this->file = $file;
            $this->data[0] = $this->decode_image_raw($file);
            $part_number = $this->ReadHeader(0,true);
            if($part_number!==0){
                throw new Exception('Not first part.');
            }
            for($i=1;$i<$this->parts;$i++){
                $filen = str_replace('0.png',$i.'.png',$file);
                $this->data[$i] = $this->decode_image_raw($filen);
            }
        }

        function ReadHeader($index,$strip=true){
            $mark = substr($this->data[$index],0,3);
            switch($mark){
                case '[I]':
                    $endof = strpos($this->data[$index],'|');
                    $header = substr($this->data[$index],0,$endof);
                    if($strip){
                        $this->data[$index] = substr($this->data[$index],$endof+1);
                    }
                    list($part_number,$this->parts,$this->efile) = explode('.',$header);
                    $part_number = hexdec($part_number);
                    $this->parts = hexdec($this->parts);
                    $this->efile = base64_decode($this->efile);
                    return $part_number;
                    break;
            }
        }

        function decode_image_raw($image){
            $ret = '';
            $gd = imagecreatefrompng($image);
            list($w,$h) = getimagesize($image);
            for($ih=0;$ih<$h;$ih++){
                for($iw=0;$iw<$w;$iw++){
                    $rgb = imagecolorat($gd,$iw,$ih);
                    $ret .= chr($rgb);
                }
            }
            imagedestroy($gd);
            return $ret;
        }

        function Extract_File(){
            $fp = fopen($this->efile,'wb');
            foreach($this->data as $d){
                fwrite($fp,$d);
            }
            fclose($fp);
        }
    }
?>

Split: 1gb which is greater than file size. I hope you did benchmark the 100mb file with 50kb splits (the code I demo'ed to begin with)

Specs: Dell Studio 15
Ram: 3gb
OS: Vista 32bit
Processor 2.1ghz

Owned.

[Hyperz]

Copy/pasted your code and hit refresh.



I might add that this kind of synthetic test is useless, mainly because image hosts have file or resolution restrictions.

[SplitIce]

Not sure whats so different between our 2 pc's. Maybe its the 1066mhz ram or the intel processor IDK. Doesnt even seem to use full cpu usage on your pc either strange.

I did the test again.



And i fixed a discrepency between the posted code and the code I have on my PC on the posted version.

[Hyperz]

I ran the script a second time and removed the line that creates the text file, now the cpu usage doubled to 50% spread over 4 cores nicely. Thinking I was gonna see a much lower time I got 194 secs, even higher. It's odd, but realistic for being php. anything below 10 seconds is a bad result for those sizes. Drawing to images ain't cheap on the CPU and PHP was just not made nor optimized for this kind of work.

Edit: Mind downloading .net 4 and running my test? I'm wondering if that would result in strange times to.

[SplitIce]

lol nah I dont want to install 4 till its stable this is my work/class laptop I cant afford to break anything on it.

BTW ran it again and got 9 seconds, and the output is correct, decodes correctly. Anyway might even be different gd versions or anything.

[Hyperz]

I'm sure it has nothing to do with ram speed or file versions, the difference is to big. The only thing that could explain this away is some extremely effective catching but then your 10 seconds would be to big.

But when I look at my 1st mockup it did a 191MB file (zip) in 277 seconds. Your code isn't optimized so the results it gives on my PC are actually rather accurate when compared to my 1st test, yours being a bit slower (again what would be expected of php). There is nothing running on my PC that could be bottle-necking the tests either.

To give an insight in why I believe the results from your script on my pc can be trusted compared to my last test (33sec/2.1MBs/70mb file) I'll explain the method:

First thing I did was switch to FastPixel. What this does is direct I/O with the in-memory byte array that holds the color data of a Bitmap object. That method alone will vastly outperform any drawing via php. Because the overhead is next to none and it doesn't cause the kernel to start using half of cpu time. Next thing I did was optimize its get/setpixel methods so that it no longer requires a color to be passed allowing me to write directly from the data buffer to the bitmap buffer -> byte to byte in memory = FAST. That also saves 2.560.000 Color objects from being created at 1600x1600px. Some other tweaks were done to FastPixel to result in less "x = new y()" statements, again saving 2.560.000 cpu intensive operations for each. All this gets executed in parallel (new threads) for each image to put all cores to work.

I think we both know that this method should outperform any php implementation by far because if php would be faster its the same as saying that php is faster than the language it was written in, C. But I'm gonna leave an open mind for your results because if it isn't some flaw I want to know why you are getting those low times. So do let me know if you can think of something.