Serious flaws in Copenhagen study that claims to show bike lanes are unsafe

We build bike lanes to make us safer and more comfortable while riding our bikes. Cities all over the world are building painted bike lanes and separated bike lanes. Knowing whether bike lanes are actually safer is important, to say the least. The science of the safety of bike lanes, however, is a bit behind.

The science doesn't have to be perfect in order for us to take some action, but it needs to be helpful. We need to understand how research was done and how the researcher came to their conclusions. When it comes to a Danish study by Danish researcher S. U. Jensen titled "Bicycle Tracks and Lanes: a Before-After Study" we probably should not trust its conclusions that bike lanes and separated bike lanes are unsafe. Or so argues Dr. Kay Teschke, with whom I corresponded by email last winter. (Photo of woman and girl cycling on Copenhagen street by Ian)

I'm looking at this study in particular because there are competing claims to what it actually proves. There are those who have argued that Jensen's study is proof that separated bike lanes (more commonly known as cycle tracks in Europe) are dangerous. But there are also those who have argued that when read properly the study actually shows that separated bike lanes are safe (see postscript below).

I decided to get to the bottom of this and contacted Dr. Kay Teschke of the University of British Columbia to find what an expert in epidemiological research--who also conducted a large research study on cycling safety--has to say about this study. I'm reaching out to Jensen as well and will post his response if I get one.

Jensen, in his Copenhagen study, came to the controversial conclusion that cycling on cycle tracks* is less safe than cycling on streets without any cycling infrastructure. Jensen concluded:

The safety effects of bicycle tracks in urban areas are an increase of about 10 percent in both crashes and injuries. The safety effects of bicycle lanes in urban areas are an increase of 5 percent in crashes and 15 percent in injuries. Bicyclists’ safety has worsened on roads, where bicycle facilities have been implemented.

Safety is worse with bike lanes? Jensen's conclusions are counter-intuitive and don't fit well with the results of a number of other studies, as was shown in a recent meta study of scientific studies of cycling and injuries. (Doing meta-analysis is common in epidemiology, where researchers compare different studies and look for patterns.) Given that Jensen's study comes to this irregular conclusion it would be easier to trust if we had a clear idea of how he arrived there. That, however, is one of the main problems with Jensen's paper: a lack of transparency.

Black box

The main issue with Jensen's study is that it's a black box; an algorithm that he never reveals. Jensen shows us the initial numbers that he measured, then puts them into his black box and out the other end comes the inverse.

Says Dr. Teschke:

Jensen did a very elaborate analysis with lots of adjustments. It is good to take into account factors that might bias unadjusted results, but usually the first analysis is the simple unadjusted one. Jensen does not present the unadjusted results, so I calculated them based on the data presented in his paper. On page 4, he indicates that the before and after periods studied were equally long. In Table 3 on page 9, he presents the observed before and after data on crashes and injuries. He also presents expected after data, based on all the adjustments. But let’s first look at the observed, before and after. In every row, except two (Crashes Property damage only and Intersections All crashes) the observed injuries or crashes after are lower than the observed before. On page 12, he indicates that there was a 20% increase in bicycle and moped traffic in the after period. So a calculation of crude relative risks (RR) for after vs. before:

  • Bicyclists and moped riders, all injuries total:
    RR = (injuries after/traffic after)/(injuries before/traffic before) = (406/1.2)/(574/1) = 0.59
  • Bicyclists and moped riders, intersection injuries:
    RR = (injuries after/traffic after)/(injuries before/traffic before) = (285/1.2)/(353/1) = 0.67
  • Bicyclists and moped riders, on links injuries:
    RR = (injuries after/traffic after)/(injuries before/traffic before) = (121/1.2)/(221/1) = 0.46

All three of the unadjusted results in the after vs. before comparison for bicyclists and moped riders show a reduction in risk (RR=1 means the same risk after and before, RR < 1 means lower risk after, RR > 1 means higher risk after). It is very strange not to report these results in the paper. They mean that over the period studied, the risk for cyclists and moped riders went down in the period after installation of the cycle tracks.

The question that should be answered with the adjustments is whether this reduction in risk is because of the cycle tracks or whether it is just a time trend - perhaps risk also went down on routes without new cycle tracks. If the comparison streets used for the adjustments were really comparable and if all the adjustment assumptions are unflawed, then the answer to that question would be “the reduction in risk is not from the cycle tracks”. But to take the conclusion further than that and say, after all these adjustments, cycle tracks are less safe (i.e., completely reversing the crude results)? This requires a level of trust in the adjustments that is very hard to justify in my view - especially given the difficult-to-follow description of the methodology and the many assumptions involved.

So can we trust Jensen's numbers? I don't think we can. Dr. Teschke's preliminary calculation of risk based on Jensen's numbers came to a Relative Risk of 0.59 compared to a higher risk of 1 for a street without bicycle facilities. In other words, Jensen's raw numbers support the conclusion that bicycle facilities reduce risk of injury. But it would be quite odd, Dr. Teschke explains, that the final, adjusted result would show the opposite of this. Yet when the final result comes out of Jensen's black box they are just that.

Science needs to be transparent and reproducible and this study falls short of that standard.

No one study can be the final word one way or the other. In the much more studied world of health and medicine, epidemiologists are looking for consensus among studies before coming to conclusions. The Copenhagen study has too many problems to serve as the final word on bike lane safety for policy makers.

Postscript

I wasn't the first blogger to question this study, I had also asked Dr. Teschke other questions about this study and how it related to what another statistician, Dr. Lon Roberts, had said about the study.

Another blogger from Texas, Jason Roberts, was also interested in understanding this controversial study better and had asked Dr. Lon Roberts for his opinion on the study. Jason linked to a simplified version of Jensen's study (called “Road safety and perceived risk of cycle facilities in Copenhagen”) that had even less information about Jensen's methodology and thus may have mislead Dr. Roberts. At least, that's my theory.

Dr. Roberts told Jason that "the Copenhagen study shows that the "likelihood an individual bicyclist will experience an accident goes down as the number of bicycle riders go up".

Furthermore Roberts says:

Using Soren’s percentages, here’s an example starting with the assumption that 10 bicyclists out of 10,000 will experience an accident over a certain period of time if there are no bike tracks:

On an individual basis, there’s a 10 out of 10,000 (or 0.1%) chance that an individual biker will experience an accident if there are no bike tracks
When the bike tracks were added, the accident rate increased by 9%. In other words, if there are 10 accidents without the tracks, the number of accidents increases 10.9 (or approximately 11). On the other hand, the number of bike riders increased by 18%, from 10,000 to 11,800. Therefore, on an individual basis the likelihood of an accident with the tracks added is now 11 out of 11,800, or 0.09%, as opposed to 0.1% without the lanes/tracks.

So Dr. Roberts is basically saying that Jensen had accounted for bicycle traffic volume when reporting the numbers (greater numbers of cyclists will always have some effect on increasing crashes/accidents). But Dr. Teschke seemed to be saying that Jensen had accounted for bike traffic. So I asked for more explanation.

I asked:

In your discussion paper you listed RR = 1.10 versus Jensen's estimates of expected injury rates. Does this mean that Jensen had taken into account bicycle traffic volume when he provides an estimate of injuries +10% (Table 3)? I'm assuming you added .10 to 1?

And if Jensen has already accounted for bicycle traffic in his estimate I'm confused about Roberts' calculation. Is Roberts' accounting for the denominator a second time?

Dr. Teschke responded:

Jensen’s formulas indicate that he did take bicycle and moped traffic into account - in more than one direction. He also took motor vehicle traffic into account. The reasoning for the latter is not clear to me. Perhaps he is saying that if MV traffic volumes went down after cycle tracks were installed, you would expect fewer crashes. But if lower MV traffic is one of the pathways to lower bike and moped crashes on cycle track routes, that is a good thing, not something to be adjusted out of the analysis.

You are right, I added .10 to 1.

Robert’s calculation is not very clear to me. It does seem to assume that Jensen did not take bicycle and moped traffic into account. The formulas in Jensen’s paper suggest he did.

But it is easy to be confused about what Jensen did. It is not clear what alpha and beta are, or how he chose the values for those parameters. He mentions “Danish crash prediction models” but does not provide a citation. Although he laid out the formulas for the traffic adjustments, he did not do the same for either the trend or regression-to-the-mean adjustments. When an adjustment method reverses the unadjusted result, it is important that the method be clear and highly defensible.

I also asked Dr. Teschke why Jensen's study wasn't included in their literature review.

Herb:

  1. Why was this one missed from the lit review? Did it not meet the criteria?

Dr. Teschke:

You are right, we didn’t include the Jensen study in our literature review, because it did not meet our criterion of being published in the peer-reviewed literature. It was published in conference proceedings not a peer-reviewed journal. After the review was published, many people noted that we did not include this study and, when we gave the reason, they argued that the reviews of the Transportation Research Board are more rigorous than those of many conferences, so it should be considered truly peer-reviewed.

We have referenced the Jensen study in subsequent publications, for example in the Discussion of our injury study.

I hope this is useful for some of you interested in connecting the dots between bike lanes, protected bike lanes and safety. It's not easy to dive into the data, but luckily scientists are taking subject matter more seriously.

Ontario Liberals promised bike infrastructure fund a drop in bucket

The Ontario Liberals promised $25 million towards cycling infrastructure this year. While this is certainly better than zero dollars and while cycling organizations such as Share the Road did good to get excited about it, I'm going to look at the gift horse in the mouth. I'm here to provide the horse droppings on the parade (or some such metaphor).

For the 2013-14 fiscal year Ontario dedicated about $6 billion towards transit and highways. The $25 million for cycling infrastructure is over three years and spread out across a population of 13.5 million. That comes out to 60 cents a person per year. Compare that to $450 per person per year for transit and highways!

The Ontario government loves cycling about 0.1% as much as they love big transit and highway projects.

I'm well aware that province-wide that cycling rates are low (though they do increase quite a bit when we include all the recreational cycling), the rate is still an order of magnitude higher than 0.1%. Toronto as a whole is around 2% but there are parts of central Toronto that are almost 20%.

While the announcement is certainly good news, it is just barely so. It will have a minor impact on transportation choices in this province. The best thing the government could do would be to pick just 2-3 big projects in cities where there is a sweet spot of a high impact on bike mode share and a willing city government to quickly implement the change. Otherwise the money might be spread too thinly to even be noticeable.

Perhaps if cyclists across this province went around our neighbourhoods and asked for 60 cents from all our neighbours we could double this tiny fund in no time.

The good news, as cycling advocates have pointed out, is that Ontario will now incorporate cycling infrastructure into all provincial highway and bridge work.

“The experience of jurisdictions where they do that is it actually doesn’t cost you any more because… you basically integrate it. You can see it on Highway 7 in Toronto: You’ll see the Viva (bus) lines, you’ll see a sidewalk and you’ll see a roadway with several lanes on it and you’ll see a cycling trail. From now on, we’ll just simply build it in like we build sidewalks unless there’s a cost reason,” he said.

The exception will be where it doesn’t make sense, such as in Brampton, where there are sidewalks along highways that aren’t being used. Those are being re-purposed as active transportation corridors.

This will have a longer term impact even though the changes will be slow to be seen since it doesn't involve actual retrofitting of highways that don't have other work scheduled. And it doesn't seem to include any policy directive for municipalities to do the same for their own infrastructure projects. Neither has the province promised to make implementing cycling infrastructure easier. Councillor Denzil Minnan-Wong has been frustrated--as have most cyclists--that the implementation of protected bike lanes on Richmond and Adelaide has dragged on for years, partly because of the onerous environmental assessment process:

The public, he said, doesn’t understand the “convoluted” environmental assessment process that means it can take four or five years to realize a project.

Let me pat you on the back Minister Glen Murray, but we've only just started.