Search This Blog


Tuesday, September 8, 2015

Ask your questions carefully

Sometimes researchers are surprised by their results. According to this article, researchers expected exercisers to overestimate the food "reward" earned through exercising. However, the subjects underestimated - by a lot. On average their estimates provided less than half of the calories they actually burned.

How can this be? After all, isn't society getting fatter and fatter in spite of exercise? (Google "we exercise but we're still fat" and see how many articles you get.) I suspect that these researchers wanted to show that we overeat following exercise because we can't estimate the calories burned/returned. Oops. Their data simply didn't support that.

However, even though their quantitative data went against their expectation, they have an explanation:

“Potentially this might be seen as encouraging, but as we pointed out in the paper, we have qualitative evidence [emphasis added] that their intentions would have been to actually eat more when the training had finished, even though they were reporting by underestimating,” Williams said.
Many participants remarked that they would have rewarded themselves with more food or drink than they had estimated would compensate, he said.

In other words, the carefully designed experiment didn't provide the expected results but they still have some evidence that exercisers overeat. If their question was really "how much will you eat after you exercise" instead of "how many calories do you think you can replace after exercise" then that's the question they should have asked.


Thursday, August 20, 2015

Really? This is the best they could do?

With the Ashley-Madison data hack in the news, a data visualization web site created a bar graph showing the cities with the highest number of AM accounts.

There are 268,171 accounts in New York City and only 86,897 in Atlanta. So what? There are a heck of a lot more people in NYC (8.406 million) than in Atlanta (447,841). It's no surprise that NYC has more Ashley-Madison users than Atlanta. I bet NYC has more XBox Live accounts too.

As one of my colleagues likes to say "Where there's a lot of stuff, there's a lot of stuff". This data is meaningless without per-capita reporting.

In New York City, 268,171 accounts out of a population of 8,406,000 is 3.2% of the population. For Atlanta, 86,897 accounts out of a population of 447,841 is a whopping 19.4% of the population.

In other words, even though New York has roughly three times as many accounts as Atlanta, Atlantans are SIX TIMES more likely than New Yorkers to be on the site.

Wednesday, September 3, 2014

You can't have it both ways

Take a look at the third "weight loss myth" on this list. It says that research (in other words, actual data) shows that there's no difference in BMI for students with and without regular PE classes. In other words, there is no evidence that PE classes help reduce childhood obesity.

Then it says "Anything that gets children moving is a step in the right direction, however...".

That's a contradiction. You can't have it both ways.  Either there's evidence that PE classes help or there isn't. You can't say research shows that PE has no effect on weight but PE classes will help anyway.

If you read a lot of news about various studies you'll see that it's a common error. When people really want to believe that X works they have a hard time accepting evidence that doesn't support their belief. You will find statements such as "There's no evidence that X helps Y but based on our theories/logic, we believe X should help Y so we should keep doing X". 

The other common response is "The evidence shows no link between X and Y but that just means we need even more X".  In other words, "My belief that X works is still correct, we can ignore the data because we were wrong about how much X we needed".

That response also shows up in the article linked: "..researchers said that PE classes were falling short and suggested a curriculum where children get more activity than classes currently offer".