[c#] CHALLENGE - Beat the time!

[jayfella]

Ok.
So Hyperz and i were discussing the finer points of multi-threading, asynchronous threads and all that story.

We decided to carry out a friendly "code-off" and see who could come up with the fastest solution based on how you concluded the best way to go about doing such a thing:

The task is simple: Perform a series of (pre-decided) http requests and time the request responses. The fastest code wins. Simple. We are also using a HTTP time-out of 5000 milliseconds, to avoid dead/slow servers from hampering the results.

URL LIST: http://www.coding.extremecoderz.com/urls.txt


Let the battle commence! In a FRIENDLY way! Please DO NOT flame if you get pwnt, this is in the interest of fun and learning.

Please Also Note: We are using a console application, not a windows Forms Application.

jayfella Reviewed by jayfella on 18th Jun 2010. [c#] CHALLENGE - Beat the time! Ok. So Hyperz and i were discussing the finer points of multi-threading, asynchronous threads and all that story. We decided to carry out a friendly "code-off" and see who could come up with the fastest solution based on how you concluded the best way to go about doing such a thing: The task is simple: Perform a series of (pre-decided) http requests and time the request responses. The fastest code wins. Simple. We are also using a HTTP time-out of 5000 milliseconds, to avoid dead/slow Rating: 5

[jayfella]

MY code:

Im not a man of the book. I just write it how i see it. This probably goes against every rule in the programming book. Still. Its fast as fuck.

Basically what im doing here is throwing async out the window. I chuck everything into one huge array and throw it at a threadpool, which houses an array of HTTP-Repquests. Simplicity ftw...?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;
using System.IO;
using System.Text;

using System.Threading;
using System.Net;

namespace ConsoleApplication1
{

    class Program
    {

        public static string[] myUrls;

        // HTTP arrays
        static HttpWebRequest[] webReq;
        static HttpWebResponse[] webResp;
        static ManualResetEvent[] resetEvents;

        static Stopwatch sw = new Stopwatch();

        static void Main(string[] args)
        {
            sw.Start();
            Console.WriteLine("starting...");
            RunTest();
        }

        public static void RunTest()
        {
            string s = "";
            using (StreamReader rdr = File.OpenText("urls.txt"))
            {
                s = rdr.ReadToEnd();
            }

            myUrls = s.Split('\n');

            // Set Min-Max Threads (setting Both the same removes time-to-create delays)
            ThreadPool.SetMinThreads(100, 100);
            ThreadPool.SetMaxThreads(100, 100);

            // Declare array Sizes Before starting...
            DeclareArrays();

            for (int i = 0; i < myUrls.Length; i++)
            {
                resetEvents[i] = new ManualResetEvent(false);
                ThreadPool.QueueUserWorkItem(new WaitCallback(DoRequest), (object)i);
            }
            
            Console.Read();
        }

        public static void DeclareArrays()
        {
            webReq = new HttpWebRequest[myUrls.Length];
            webResp = new HttpWebResponse[myUrls.Length];
            resetEvents = new ManualResetEvent[myUrls.Length];
        }

        public static void DoRequest(object o)
        {
            int i = (int)o;
            try
            {
                webReq[i] = (HttpWebRequest)WebRequest.Create(myUrls[i]);
                webReq[i].Proxy = HttpWebRequest.GetSystemWebProxy();
                webReq[i].ServicePoint.Expect100Continue = false;
                webReq[i].AllowAutoRedirect = true;
                webReq[i].KeepAlive = false;
                webReq[i].UserAgent = "Mozilla Firefox";
                webReq[i].Method = "GET";
                webReq[i].Timeout = 5000;
                webReq[i].ContentType = "application/x-www-form-urlencoded";

                // Gimme some sugar!
                webResp[i] = (HttpWebResponse)webReq[i].GetResponse();
                
                Console.WriteLine(sw.Elapsed + " - Finsihed: " + myUrls[i]);
            }
            catch
            {
                // In case of fail, let the user know.
                Console.WriteLine(sw.Elapsed + " - FAILED: " + myUrls[i]);
                resetEvents[i].Set();
                return;
            }

            // All Done. Tell the ThreadPool its finished so it can be re-used.
            resetEvents[i].Set();

        }
    }
} 

MY RESULT: 14.968 seconds.

[Hyperz]

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApplication1
{
    class Program
    {
        public static Stopwatch sw = new Stopwatch();

        static void Main(string[] args)
        {
            Console.WriteLine("Starting");
            ServicePointManager.Expect100Continue = false;
            Thread.CurrentThread.Priority = ThreadPriority.AboveNormal;
            ServicePointManager.DefaultConnectionLimit = 4;

            RunTest();
            
            Console.Read();
        }

        public static void RunTest()
        {
            string filename = "urls.txt";
            string[] urls = File.ReadAllLines(filename);
            sw.Start();
            foreach (string url in urls) DoRequest(url);
        }

        public static void DoRequest(string url)
        {
            var thread = new Thread(state =>
            {
                var req = (HttpWebRequest)WebRequest.Create(state as String);
                req.AllowAutoRedirect = false;
                req.Timeout = 5000;
                req.Proxy = null;
                req.KeepAlive = false;
                req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

                try
                {
                    using (var resp = (HttpWebResponse)req.GetResponse())
                    {
                        resp.Close();
                        //req.Abort();
                        Console.WriteLine("{0} - Finished: {1}", sw.Elapsed, resp.ResponseUri.OriginalString);
                    }
                }
                catch
                {
                    req.Abort();
                    Console.WriteLine("{0} - FAILED: {1}", sw.Elapsed, req.RequestUri.OriginalString);
                }
            });

            thread.Priority = ThreadPriority.Highest;
            thread.IsBackground = true;
            thread.Start(url);
        }

        /*public static void DoRequest(string url)
        {
            var req = (HttpWebRequest)WebRequest.Create(url);
            req.AllowAutoRedirect = false;
            req.Timeout = 5000;
            req.Proxy = null;
            req.KeepAlive = false;
            req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

            var async = req.BeginGetResponse(new AsyncCallback(result =>
            {
                var request = (HttpWebRequest)result.AsyncState;

                try
                {
                    using (var resp = (HttpWebResponse)request.EndGetResponse(result))
                    {
                        resp.Close();
                        request.Abort();
                        Console.WriteLine("{0} - Finished: {1}", sw.Elapsed, resp.ResponseUri.OriginalString);
                    }
                }
                catch
                {
                    request.Abort();
                    Console.WriteLine("{0} - FAILED: {1}", sw.Elapsed, request.RequestUri.OriginalString);
                }
            }), req);

            ThreadPool.RegisterWaitForSingleObject(async.AsyncWaitHandle, new WaitOrTimerCallback((state, timeout) =>
            {
                var request = (HttpWebRequest)state;
                if (timeout)
                {
                    request.Abort();
                    Console.WriteLine("{0} - FAILED: {1}", sw.Elapsed, request.RequestUri.OriginalString);
                }
            }), req, 5000, true);
        }*/
    }
} 

RESULT: 5.183 seconds.


Using the propper async way, which is less resource intensive but slow I get around 17 seconds. That would be the part in the code that's commented out.

F# version:

open System
open System.Collections.Generic
open System.Diagnostics
open System.IO
open System.IO.Compression
open System.Linq
open System.Net
open System.Text
open System.Threading
open System.Threading.Tasks

stdout.WriteLine "Starting."

let urls = File.ReadAllLines "urls.txt" |> List.ofArray
let sw = new Stopwatch()

ServicePointManager.Expect100Continue <- false

GC.Collect();

ServicePointManager.DefaultConnectionLimit <- 480
ServicePointManager.MaxServicePoints <- 480

ThreadPool.SetMaxThreads(480, 480) |> ignore
ThreadPool.SetMinThreads(450, 450) |> ignore

let DoRequest url = async {
    try
        let req = WebRequest.Create(url:string) :?> HttpWebRequest
        req.AllowAutoRedirect <- false
        req.Timeout <- 5000
        req.Proxy <- null
        req.KeepAlive <- false
        req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
        use! rsp = req.AsyncGetResponse()
        rsp.Close()
        String.Format("{0} - Finished: {1}", sw.Elapsed, url)
        |> stdout.WriteLine
    with _ ->
        String.Format("{0} - FAILED: {1}", sw.Elapsed, url)
        |> stdout.WriteLine
    
    return true }

sw.Start()

urls
|> List.map DoRequest
|> Async.Parallel
|> Async.RunSynchronously
|> ignore

stdout.WriteLine "Done."
stdin.Read() |> ignore

I can't test this one because the timeout isn't working. Should it work it would be around 5 - 7 seconds.

[el_jentel1]

I don't know if it's Hyperz pwned Jay or F# pwned C#. Either way, I win.

[jayfella]

haha

hang on

[Whoo]

Isn't the speed of your internet connection the big unknown factor in this?

[Hyperz]

We test both programs on our pc .

[jayfella]

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;
using System.IO;
using System.Text;

using System.Threading;
using System.Net;

namespace ConsoleApplication1
{

    class Program
    {
        public static string[] myUrls;
        static HttpWebRequest[] webReq;
        static HttpWebResponse[] webResp;
        static ManualResetEvent[] resetEvents;
        static Stopwatch sw = new Stopwatch();

        static void Main(string[] args)
        {
            Console.WriteLine("starting...");
            RunTest();
        }

        public static void RunTest()
        {
            string s = "";
            using (StreamReader rdr = File.OpenText("urls.txt"))
            { s = rdr.ReadToEnd(); }

            myUrls = s.Split('\n');

            ThreadPool.SetMinThreads(512, 512);
            ThreadPool.SetMaxThreads(512, 512);

            DeclareArrays();
            sw.Start();

            for (int i = 0; i < myUrls.Length; i++)
            {
                resetEvents[i] = new ManualResetEvent(false);
                ThreadPool.QueueUserWorkItem(new WaitCallback(DoRequest), (object)i);
            }
            
            Console.Read();
        }

        public static void DeclareArrays()
        {
            webReq = new HttpWebRequest[myUrls.Length];
            webResp = new HttpWebResponse[myUrls.Length];
            resetEvents = new ManualResetEvent[myUrls.Length];
        }

        public static void DoRequest(object o)
        {
            int i = (int)o;
            try
            {
                webReq[i] = (HttpWebRequest)WebRequest.Create(myUrls[i]);
                webReq[i].Proxy = HttpWebRequest.GetSystemWebProxy();
                webReq[i].Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
                webReq[i].ServicePoint.Expect100Continue = false;
                webReq[i].AllowAutoRedirect = false;
                webReq[i].KeepAlive = false;
                webReq[i].UserAgent = "Mozilla Firefox";
                webReq[i].Method = "GET";
                webReq[i].Timeout = 5000;
                webReq[i].ContentType = "application/x-www-form-urlencoded";

                
                webReq[i].AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

                using (webResp[i] = (HttpWebResponse)webReq[i].GetResponse())
                {
                    // webResp[i].Close();
                    Console.WriteLine(sw.Elapsed + " - Finsihed: " + myUrls[i]);
                    resetEvents[i].Set();
                    return;
                }
            }
            catch
            {
                // webResp[i].Close();
                Console.WriteLine(sw.Elapsed + " - FAILED: " + myUrls[i]);
                resetEvents[i].Set();
                return;
            }
            
        }
    }
} 

[Ronald]

Hyperz you tryhard!

[Zeokat]

On jayfella?s code, exists one line not needed using GET method:

webReq[i].ContentType = "application/x-www-form-urlencoded";

Content type is used mostly with POST method... not needed on GET requests.

Anyways... speed results can be confused. All depends on how many timeouts the day of the test, the speed and the load of the server the day of the test.... etc...

Anyways are good pieces of code.