This is part of my Everyday Coffee series, in which I review five affordable SCAA-certified coffeemakers. I’m in the middle of testing one of the coffeemakers this week, so here’s a look at how I actually do my tests. Read on to find out why subjective is the only way to go.

I Love Coffee

And there’s no better way to make coffee than by hand. A manual brewing method, such as a pourover or French press, will always give you the best results. And taste aside, the act of handcrafting my own coffee is just therapeutic.

Unfortunately, I don’t always have the time to babysit a manual brewing process. On workdays, I like — no, NEED — to capture every last minute of sleep in the mornings. Shaving off the 15-20 minutes it takes to make two pourovers is worth every second of sweet, sweet dreams.

Programmable coffeemakers are one solution. But most home coffeemakers, to use the technical term, suck. For a whole bunch of reasons, the coffee comes out underextracted and weak. Fortunately, there’s a new generation of high-end, SCAA-certified coffeemakers out there that deliver a really great cup of joe.

But here’s the catch: coffee is subjective. This is a problem because most of these coffeemakers aren’t cheap. Unless you’re willing to buy a half dozen coffeemakers and put them through their paces side by side, it’s really difficult to find the one that matches your taste when you’re looking at technical specifications. Side-by-side comparisons of feature lists and tech specs just doesn’t cut it.

Setting Up A Fair Test

Each coffeemaker was put through a minimum of 14 cycles. Unless otherwise noted, I used the “golden ratio” of coffee: 57 grams of whole beans (weighed before grinding) to one liter of water. All of the beans were ground using my Baratza Encore grinder on the #15 setting.

For water, I used room-temperature bottled water from Roaring Spring. If the reservoir was filled directly from the water cooler, the water was allowed to sit for 15 minutes before brewing. This gives the water the same starting temperature for each test — more or less.

I used three different sources of coffee. I used a variety of specialty-roast beans from a local roaster (Little Amps, Elementary, or St. Thomas) to see how the units handled subtle, delicate flavors. I used French Roast beans from Starbucks (beans roasted in large batches give consistency) to see how the units handled bold flavors.

And as a final test, I brewed one pot using my special “torture blend”: preground Starbucks Sumatra that was purchased at a local supermarket and spread out on a plate to air for 7 days. The point of this is to see if any of the coffeemakers are capable of breathing new life into abused coffee.

Because let’s face it: your aunt Edna sees nothing wrong with using coffee that’s been in the freezer since 1987.

Weekday morning brews were performed using the unit’s built-in timer using the unit’s maximum capacity. Each timer was set to allow the coffee to sit on the warmer for about 20 minutes before being served. This helps prove or disprove some manufacturer’s claims to “not scorch the coffee”. Afternoon and weekend brews were done on demand, typically on a 6-cup setting, and consumed immediately after the brew cycle finished. Each brew cycle used the manufacturer’s recommended or default settings unless otherwise noted.

Time was measured from the moment I pressed “start” until the moment the unit indicated it was done. If the unit had no such indication, I considered the pot “done” when the flow was reduced to one drip per second.

Why You Should Trust Me

Google any of the coffeemakers I’m testing. You’ll find a half dozen pages of reviews, nearly all of which heap glowing praise on the test units. You’ll also find sites that say things like “we received this unit for free, but we super-pinky-promise that didn’t affect our review at all”.

Read the site a little more, and you’ll discover that they smother nearly everything they test with lavish praise. If they’re really good, they’ll even throw out a few inconsequential downsides. “The coffee was great, but the color was the wrong shade of black”. That sort of thing.

The problem is, not every product is that great. It really makes you wonder how these “unbiased” reviewers have somehow managed to run into nothing but flawless product after flawless product, day in and day out, every single time.

You’ll have to decide for yourself whether they’re really “unbiased” or not.

Every single coffeemaker I’m testing was paid for out of my own pocket. So were the beans and the water. None of my pages go to affiliate links, and I don’t get a commission on anything you buy. The only relationship I have with any of the companies or products mentioned in this series is “I’m their customer, and I spent a ridiculous amount of money putting these things through their paces.”

Most of us work hard for our money, so you should never have to question whether a reviewer has your best interests at heart. You’re getting my honest, un-compensated opinion: good, bad, or ugly.

Everything that Matters is Flavorful

Here’s the bottom line: taste is king. If it doesn’t taste good, it’s not worth your money.

Coffee purists will argue that what really matters is the total dissolved solids in the cup. They’ll talk about the pH level, saturation time, flow-through rate, and turbulence. They’ll probably go off on a tangent arguing whether a pourover or French press is superior (hint: they’re both excellent tools that accomplish very different tasks). And they’re technically correct — all those things matter.

But at the end of the day, what do all of those factors lead to? Taste.

Purists aren’t much fun to drink with anyway.