I finally decided on a formula for it. Before I post the list, I should explain the intent of this different formula.

I was looking at things like Metacritic and Mega-critic and thinking if you base the rating on just a weighted average, it gives more power to extreme reviews, especially extreme negative reviews. So if everybody loved an album but one publication really hated it, it could drag it down several points. So the idea behind this formula is to equalize the power of everyone's vote.

What I do is, for each pair of albums that at least ten publications have reviewed both of, check which one more of them think is superior. That one gets a 'win'.

Then, the list is compiled from the bottom up, lopping off the lowest point total every round. Except every round, points are recalculated as such:
Win against album still in: 1 point
Tie against album still in: 0.5 points
Win against album already out: 0-0.5 points depending on how far down the list it is.
(So, in the round determining 8th place, a win against 9th place gets 0.5 points and win against last place gets just above 0.)

Counting losses the same as not having enough reviews in common also has the effect of boosting albums that've been reviewed more.

Right now I'm up to 33 albums, and I've decided every day to add the highest rated one on metacritic with at least 15 reviews that hasn't been added yet.

All that being said, here's the top ten I came up with (With anything that will rocket up the list when it gets more reviews also added).

Position. Band - album Win-loss-tie Percent of individual rating comparisons won
1. Vampire Weekend - Vampire Weekend 28 - 3 - 1 0.6226053639846744
2. Drive By Truckers - Brighter Than Creation's Dark 24 - 2 - 4 0.5869074492099323
3. Bon Iver - For Emma, Forever Ago 20 - 0 - 0 0.6878787878787879
4. Los Campesinos! - Hold On Now, Youngster 17 - 2 - 1 0.6222910216718266
5. Nick Cave - Dig Lazarus, Dig 14 - 0 - 2 0.6777408637873754
6. British Sea Power - Do You Like Rock Music? 17 - 8 - 4 0.5267665952890792
7. Black Mountain - In The Future 18 - 12 - 0 0.5162271805273834
8. Hercules and Love Affair - Hercules and Love Affair 14 - 0 - 0 0.7585616438356164
9. The Gutter Twins - Saturnalia 14 - 9 - 2 0.5037688442211056
10. Hot Chip - Made In The Dark 13 - 15 - 4 0.47289156626506024

20. Shelby Lynne - Just A Little Lovin 8 - 4 - 0 0.5260223048327137
22. DeVotchKa - A Mad & Faithful Telling 6 - 1 - 4 0.577490774907749
27. Erykah Badu - New Amerykah: Part One(4th World War) 6 - 0 - 2 0.6489795918367347

Very interesting! It seems like Henrik's formula a little in that it compares ratings album per album on a win/lose basis. Well done!

I noticed a small mistake in my code after I put up the results that made it so a win over any album already out was given .5 points instead of between 0 and .5 depending on position.

When I fixed that error, the list came out like this: (Plus I added in a statistic that tells you how many of my 30 review sources gave it a rating.)

1. Vampire Weekend - Vampire Weekend 32 - 3 - 0 0.6356655290102389 26/30
2. Bon Iver - For Emma, Forever Ago 26 - 0 - 1 0.6950261780104712 16/30
3. Drive By Truckers - Brighter Than Creation's Dark 26 - 2 - 4 0.5909090909090909 22/30
4. Nick Cave - Dig Lazarus, Dig 22 - 2 - 1 0.650375939849624 17/30
5. Los Campesinos! - Hold On Now, Youngster 19 - 3 - 1 0.6372282608695652 15/30
6. Hercules and Love Affair - Hercules and Love Affair 16 - 0 - 0 0.7608024691358025 14/30
7. British Sea Power - Do You Like Rock Music? 23 - 9 - 2 0.5465779467680608 23/30
8. DeVotchKa - A Mad & Faithful Telling 13 - 3 - 2 0.5611620795107034 12/30
9. Why? - Alopecia 13 - 1 - 1 0.632890365448505 12/30
10. Black Mountain - In The Future 21 - 12 - 1 0.526595744680851 25/30

And as for which 30 review sources I chose, basically I picked the ones on Metacritic that seemed to give the most reviews and seemed to give a fair listen to the most different genres.

The only ones on Metacritic I absolutely refuse to use are: Rolling Stone, Q Magazine, Billboard, Entertainment Weekly, Amazon, RapReviews.com. Anything else I'd consider switching in if one of my sources either seemed too genre-biased or didn't give enough reviews.