I just read Doublas Eadline's post at  Linux Magazine and found it really fascinating. The best tidbit to chew on is:

This is where all the hardware excitement meets the cold reality of parallel programming. We are all familiar with “Moore’s Law” (the transistor density doubles every two years). Many people have probably not heard of “May’s Law.” (for the purist, you can substitute “trend” for “law”). In any case, David May states his law as follows, “Software efficiency halves every 18 months, compensating for Moore’s Law.” Think about it. Every new generation of hardware

Moore's Law - The number of transistors on a chip doubles roughly every 2 years

May's Law - Software efficiency halves every 18 months, compensating for Moore’s Law.

Chew on that.

I had previously installed CUDA and CUDA SDK on my computer running Ubuntu 10.04. Matlab is also installed on the computer, and some of the students are currently using libSVM for data classification research. One of the major problems that we are having is that the data sets are so large that libSVM is simply too slow to use. The solution - let's add cuSVM to the installation so they can speed up their code easily.

I planned on following the instructions here where the first step is "Download Matlab plug-in for Cuda (Version 1.1) from http://developer.nvidia.com/object/matlab_cuda.html". Along the way I decided that I would not only download the Matlab plug-in for Cuda, but that I would install and test it as well. Though, along the way I ran into a couple of issues. Here are the steps that I followed:

1. Download the Matlab plug-in for CUDA and extract it
2. Open the Makefile for editing
3. Add a line at the very beginning of the file that specifies the current directory. In my example this is:

   CWD=/path/to/my/home/directory/Matlab_Cuda_1.1

4. If you are on a 64-bit system (as I am), make sure that line 6 reads as :

   INCLUDELIB  = -L$(CUDAHOME)/lib64 -lcufft  -Wl,-rpath,$(CUDAHOME)/lib64

5. Change line 11 (export Matlab=/usr/local/matlab) so that it matches your Matlab installation. In my case this was /opt/Matlab
6. On lines 38, 45, and 52 you will find references to a script called nvopts.sh. When I first ran this Makefile, I continually got errors telling me that nvopts.sh did not exist. How to fix this? Add the correct path to the nvopts.sh reference. I did this by following the instructions in step 3 and then I adjusted lines 39, 46, and 53 to read as follows:

   $(NVMEX) -f $(CWD)/nvopts.sh $(INCLUDEDIR) $(INCLUDELIB)

7. Save the Makefile
8. Open the file called nvmex
9. Comment out lines 1448-1454. Just in case I have the line numbers slightly wrong, these lines read like this:

   if [ "$compile_only" != "1" ]; then
       if [ "$gateway_lang" = "C" ]; then
       	files="$files $MATLAB/extern/src/mexversion.c"
       else
           files="$files $MATLAB/extern/lib/$Arch/version4.o"
       fi
   fi

   and should now look like

   #if [ "$compile_only" != "1" ]; then
   #    if [ "$gateway_lang" = "C" ]; then
   #    	files="$files $MATLAB/extern/src/mexversion.c"
   #    else
   #        files="$files $MATLAB/extern/lib/$Arch/version4.o"
   #    fi
   #fi

10. Save nvmex
11. In the case that your paths are not set properly (mine are not) open up nvopts.sh. Make sure that the call to nvcc on lines 56, 87 and 164 are set correctly. I changed these from

    CC='nvcc'

    to

    CC='/usr/local/cuda/bin/nvcc'

12. Save nvmex
13. Open the terminal in the directory where your Makefile is and type 'make all'. Enjoy!

An interesting article has been posted over at Linux Magazine. The article interests me mainly because it gets to the heart of an issue that I myself ponder over: As hardware overtakes software, how will software catch up? This is a very important question, especially in the HPC world as there is no single language that elegantly translates across multiple architectures.


Read the rest of the article here.

This article is very interesting. A very hot topic in parallel computing these days is power consumption and this is the first that I've heard of the parallel power law. I'm intrigued.

While I never participated in the Intel Threading Challenge, I still find the problems really intriguing. Why? Because they are problems designed to test threaded development which is not only cool but is also going to play a large part in the future of computing. I would call all this problem for the most part concurrent, not parallel. Why? Read this and you'll understand completely. Now, on to the challenge!

Problem # 1 states:

Problem description: Given a set of unsorted items with keys that can be considered as a binary representation of an integer, the bits within the key can be used to sort the set of items. This method of sorting is known as Radix Sort.

Write a program that includes a threaded version of a Radix Sort algorithm that sorts the keys read from an input file, then output the sorted keys to another file. The input and output file names shall be the first and second arguments on the command line of the application execution.

The first line of the input text file is the total number of keys (N) to be sorted; this is followed by N keys, one per line, in the file.  A key will be a seven-character string made up of printable characters not including the space character (ASCII 0x20). The number of keys within the file is less than 2^31 - 1.  Sorted output must be stored in a text file, one key per line.

Timing: If you put timing code into your application to time the sorting process and report the elapsed time, this time will be used for scoring.  If no timing code is added, the entire execution time (including time for input and output) will be used for scoring.

Example Input file:
8
H@skell
surVEYs
sysTEMS
HASKELL
Surveys
1234567
SURveys
systEMS

Example Output file:
1234567
H@skell
HASKELL
SURveys
Surveys
surVEYs
sysTEMS
systEMS

My solution (both serial and parallel):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Collections;
using System.Diagnostics;
using System.Threading;
using System.IO;
 
namespace RadixSort {
 
    class Program {
        static void Main(string[] args) {
            StreamReader sr;
            int length;
            TimeSpan serial, parallel;
            sr = File.OpenText(@"C:\Documents and Settings\rgreen\Desktop\Threading\Threading\rsTestK100.dat");
 
            length = Convert.ToInt32(sr.ReadLine().Trim());
            string[] values = new string[length];
            string[] newValues = new string[length];
            for(int x = 0; x < length; x++) {
                values[x] = sr.ReadLine().Trim();
            }
 
            Stopwatch sw = new Stopwatch();
 
            //
            // Serial
            //
            sw.Start();
            RadixSort(values).CopyTo(newValues, 0);
            sw.Stop();
            serial = sw.Elapsed;
 
            //
            // Parallel
            //
            sw.Reset();
            sw.Start();
            ParallelRadixSort(values).CopyTo(newValues, 0);
            sw.Stop();
            parallel = sw.Elapsed;
 
 
            Console.WriteLine("Serial Time: " + serial);
            Console.WriteLine("Parallel Time: " + parallel);
            Console.ReadLine();
        }
 
        public static string[] ParallelRadixSort(string[] array) {
            int length = array.Length;
 
            Parallel.For(0, array[0].Length - 1, delegate(int curRadix) {
                int index = array[0].Length - 1 - curRadix;
                array = new MergeSort(array, array[0].Length - 1 - index).Results;
            });
 
            return array;
        }
 
        public static string[] RadixSort(string[] array) {
 
            int length = array.Length;
 
            for(int curRadix = array[0].Length - 1; curRadix >= 0; curRadix--) {
                array = new MergeSort(array, array[0].Length - 1 - curRadix).Results;
            }
 
            return array;
        }
    }
}

And there you have it!

Project Euler problem #6 is

Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.

When I first began looking at this problem I wanted to experiment a bit with the Parallel Extensions for .NET, so I started where any parallel algorithm begins: with a sequential algorithm. The algorithm here is rather trivial: Loop over the natural numbers from 1 to 100. In order to sum the squares I will sum the square of each number. In order to square the sum I will sum the numbers and then square them. The difference is the answer. So, first, I came up with two functions: SumOfSquares and SquareOfSums.

My first attempt at these functions ended up something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

static double SumOfSquares(int min, int max) {
    double result = 0;
    for(int x = min; x <= max; x++) {
        result += Math.Pow(x, 2);
     }
     return result;
}
static double SquareOfSums(int min, int max) {
    double result = 0;
    for(int x = min; x <= max; x++) {
        result += x;
    }
    return Math.Pow(result, 2);
}

Both of those functions are very straight forward. My first thought after writing these was, "Hey, why not use some LINQ?" So I did. Here is how the functions change:

1
2
3
4
5
6

static double LinqSumOfSquares(int min, int max) {
    return Enumerable.Range(min, max).Select(d => Math.Pow(d, 2)).Sum();           
}
static double LinqSquareOfSums(int min, int max) {
    return Math.Pow(Enumerable.Range(min, max).Select(d => d).Sum(), 2);    
}

Wow! Talk about incredible, shrinking functions! I love finding a way to make code more succinct, readable, and elegant and these changes seem to have hit the nail on the head! Anyways, that is basically all the pieces for the sequential algorithm. All you have to do is call each of those functions and take the difference. Simple, huh? But the real question is how can we parallelize these bad boys? I have a few thoughts.

1. Call each function in parallel. In other words let the sumOfSquares and SquareOfSum function run at the same time in different threads.
2. Parallelize each function invidvidually. In other words leverage the loops inside of each function in order to parallelize them.

So let's take a look at each method using the Parallel Extensions for .NET. The first method is calling each function in parallel. My first thought here was to use Parallel.Invoke in order to call each function at the same time. A little further research quickly revealed that Parallel.Invoke cannot return any values. My initial response to that: "Well that sucks." Luckily there's another class in the Parallel library called Futures. What's a future? From DevX -

"In TPL terms, a Future is basically a task that returns a value. It's like a deferred function. You start it running and then use its value later. If the Future hasn't finished calculating its value by the time you need it, it makes you wait while it finishes."

Sounds good to me. So how do we use futures? Like this:

1
2
3
4
5
6
7
8

static long Problem6Futures(int min, int max) {
 
    Future<double> fSumOfSquares = Future.Create(() => SumOfSquares(min, max));
    Future<double> fSquareOfSums = Future.Create(() => SquareOfSums(min, max));
    double result = fSquareOfSums.Value - fSumOfSquares.Value;
 
    return result;
}

Easy, huh? All you have to remember is that Futures are deferred functions that return values. The result of Future operation gets stored in an object of type Future and the value is stored in Object.Value. So how about the second method of parallelizing this algorithm? Well, it's even easier because of PLINQ - or Parallel LINQ. Let's see what it looks like.

1
2
3
4
5
6

static double ParallelSumOfSquares(int min, int max) {
    return Enumerable.Range(min, max).AsParallel().Select(d => Math.Pow(d, 2)).Sum();
}
static double ParallelSquareOfSums(int min, int max) {
    return Math.Pow(Enumerable.Range(min, max).AsParallel().Sum(), 2);
}

Hah! Even easier! All that I did was add .AsParrallel() to our data. That tells LINQ to do the processing in parallel!

So, there's Project Euler Problem #6 for you. Was it a hard problem? Not really. Are you going to see major performance results through the parallelization of this algorithm? No. But you gotta' start somewhere when you're learning how to parallelize algorithms using a new library. Today I used Futures and PLINQ and that sound's like a pretty solid start to me.

Project Euler #3 is :

Find the largest prime factor of a composite number.

My first attempt at this whipped up some typical code that simply brute forced my way to the solution. My code looked like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

static void Problem3() {
    long n = 600851475143;
    int factor = 2;
    int lastFactor = 1;
 
 
    if(n % 2 == 0) {
        lastFactor = 2;
        n = n / 2;
        while(n % 2 == 0) {
            n = n / 2;
        }
     } else {
         lastFactor = 1;
     }
     factor = 3;
 
     double maxFactor = Math.Sqrt(n);
     while(n > 1 && factor <= maxFactor) {
          if(n % factor == 0) {
              n = n / factor;
              lastFactor = factor;
              while(n % factor == 0) {
                  n = n / factor;
              }
              maxFactor = Math.Sqrt(n);
          }
          factor += 2;
      }
 
      if(n == 1) {
          Console.Write(lastFactor.ToString() + " ");
      } else {
          Console.Write(n.ToString() + " ");
      }
}

Not very elegant, but it works. So I set out to find something a bit prettier and I came across I LINQ solution here. All I did was tack on the .AsParallel() in order to give it a little speed boost. The code looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

static void LinqProblem3() {
    long largeNumber = 600851475143;
    var allPrimeFactors = from p in Primes.PrimeFactors(largeNumber).AsParallel()
                                 orderby p descending
                                 select p;
 
    foreach(var f in allPrimeFactors){
        Console.WriteLine(f);
    }
}
 
public static class Primes {
 
    // Find all prime factors.
    public static IEnumerable<int> PrimeFactors(long number) {
        // Start by removing the lowest prime (2)
        return MorePrimeFactors(number, 2);
    }
 
    // This recursive method finds all prime factors.
    private static IEnumerable<int> MorePrimeFactors(long number, int factor) {
        // Find the next prime factor
	while(number % factor != 0)
	    factor++;
	// Return it.
	yield return factor;
 
	// look again...
	if(number > factor)
		// recursively look for this factor again, using Num/factor
		// as the new big number
		foreach(int factors in MorePrimeFactors(number / factor, factor))
			yield return factors;
	}
}

Available here

As of this moment, I have worked through section 4.5.2 of the very well written Yet Another Haskell Tutorial. What have I learned so far?

1. How to use HUGS and GHC to play with and compile Haskell.
2. The basics of arithmetic, pairs, tuples, and lists.
3. The basic built in functions (map, foldr, foldl, etc.)
4. How to create source files
5. How Haskell treats types

What impressions do I have this far? Functional programming seems to be a pretty good deal in my eyes. The most appealing aspect is its base in real mathematics. All computer science algorithms are obviously based in some sort of mathematics, but with functional programming it seems that the good kind of mathematics becomes more apparent. This would be the type of mathematics that is translated almost directly to code with no worry about writing loops and other structures to manage your data.  A quick example of this would be the Fibonacci Sequence. Mathematically this looks like this:

X₁ = 1
X₂ = 1 + X₁
X_n = X_n-1 + X_n-2

In Haskell it looks like:

fib 1 = 1;
fib 2 = 1 + fib(1);
fib x = fib(x-2) + fib(x-1);

or

1
2
3
4

fib n =
  if n == 1 || n == 2
    then 1
    else fib (n-1) + fib (n-2)

As you can see above, we have simply translated a recursive mathematical formula almost directly into code! That is powerful! Another functional programming feature used is pattern matching. Notice the fib 1, fib 2, and fib n from the first example? Whenever the code encounters a call to the fib function, it matches the parameter against 1,2, or n in order to determine which function to call. Sweet!

Maybe it's just me, but the sheer mathiness of Haskell and functional programming makes me giddy! It makes me imagine a world where code is written correctly because it follows mathematics! Now that is exciting.

I'm telling you it's going to be functional programming. Myself, I am going to begin exploring Haskell, Erlang, and F# and I hope to document all of my little exploits and such here. I'll learn a bit and share a bit. If I get really ambitious you may also see some F# code pop up here as well.