Epi in the News

A very interesting read indeed. Here are my first thoughts:

1- This was an intervention study with 2 arms: one receiving the intervention (tax-free $500 per month) and one control arm (no money). It was hard to say if it was randomized or matched but I do not think it was blinded

2- Population= residents of California
Intervention= tax-free $500 per month for 2 years
Comparison= control group not receiving any money per month
Outcomes= getting full-time jobs and paying off debt. Mental wellness appears to be a secondary outcome
California is a big city in a high-income country with access to a number of amenities such as education, health, etc. I believe the results may be generalizable to other high-income countries but maybe not to low- and middle-income countries.

3- At the start of the trial, there is a marked difference in a very important potential confounding study variable which is the proportion of persons with a full-time job in each study arm: 28% of those in the intervention arm had full-time jobs compared to 32% in the control group.

4- I do not know what "statistically significant improvements" in emotional health means, nor how it was ascertained in this study.

5- This study does not demonstrate a causal effect in my opinion. Researchers did not significantly exclude the role of:
- chance: I do not see reported any confidence intervals in differences between groups, p-values, nor sample size or power calculations to detect an effect
-bias: I do not see clear participant inclusion and exclusion criteria, randomization and allocation concealment procedures, loss to follow up rates in groups to control selection bias. I also do not see how full definitions of the outcomes and how they were ascertained, blinding of data collectors, etc. to control information bias.
-confounding is not adequately discussed, measured and controlled for in this study. We do not know if randomization was done.

6- "The researchers said that the extra $500 per month was enough for people with part-time jobs to take time off so they could interview for full-time jobs that offered better pay. They also said the money could have helped people who weren’t working at all find jobs by allowing them to pay for transportation to interviews. "

1. This is an Intervention trial. It’s unclear whether there was randomisation from the article.

2. Population; People who lived in census tracts at or below the city’s median household income of $46,033.
Intervention; $500 per month for two years without rules on how to spend it (universal basic income)
Comparison; Control group who did not get the money from the same socioeconomic background although there seems to be a discrepancy in levels of full-time employment between the control and intervention group (32% of the control group were in full-time employment while only 28% of the intervention group were in full-time employment)
Outcome; To lift people out of poverty with a guaranteed monthly income. Enable people to pay off debt. Facilitate getting full-time jobs. Also improved “emotional health”.

As regards generalisability, I’m unsure who the reference population is supposed to be. Having “Universal” in the title of the intervention indicates that the reference population could be anyone in California or even in the USA. However, the program specifically included people who lived in households that were at or below the city’s median household income so perhaps that is the reference population. To enable generalisation, the study participants should be a random sample of the reference population. The article doesn’t give any information about how the participants were sampled. Without a clear definition of the reference population or the sampling methods, I don't think this study is generalisable to other populations.

3. 4% difference in rates of fulltime employment is pretty large. Was this a good choice of control group?

4. It’s unclear from the article what statistically significant improvements in emotional health means. I presume that they quantitatively assessed mental health before and after the intervention using a verified psychiatric scale and analysed their results statistically perhaps using a t-test but there are no details provided.

5. Although this is an intervention trial, which could generate strong causal evidence if designed properly, there seems to be a number of issues with the design of this study.

It’s unclear how the control and intervention groups were chosen. Was there randomisation or blinding? It sounds like researchers were able to follow up on how participants spent the money allocated over the trial period, did they know who these participants were or was that anonymised? Without randomisation, there could be confounding of the association. Lack of blinding can introduce bias.

Secondly, there are differences between the control and intervention group in terms of the level of full-time employment (differ by 4%) at the start of the programme. Given that improving employment levels was one of the outcomes this is problematic. I would like to know were there other differences in composition between the control and intervention group that could introduce further bias.

Lastly without further information, I can’t assess whether this association could be due to chance. Would need to see further statistics including p-value and 95% CIs.

Chance/bias/confounding could have affected the study results so I don't think a causal effect has been demonstrated.

6. The researchers said that the extra $500 per month was enough for people with part-time jobs to take time off so that they could interview for full-time jobs that offered better pay. They also said the money could have helped people who weren’t working at all find jobs by allowing them to pay for transportation to interviews.

Adding some of my thoughts...
1) It's a quasi-experimental study (which essentially means there is no randomization) as the investigators do control who gets the intervention and who does not, based on certain cut-offs and not randomly (here below a particular income). These studies are usually not blinded- which is the same here.. everybody knows who is in the intervention group.

2) Population: People who lived at or below the city's median household income
Intervention: 500 dollars per month for 2 years, no restrictions on how to utilize the money
Comparison: Control group from the same cohort who did not receive the money
Outcome: before and after measurements of rate full-time employment, rate of clearing off debts, improvement in emotional health

The study has been done only in the population that is below the median household income. So it cannot be generalized to the remaining population who are above this mark

3) I agree its a vague question. 28 and 32 % is the baseline rate of full employment in intervention vs control groups- which is quite similar. Only 1/3rd of this population have a full time job.

4)"statistically significant improvements" refers to the fact that the results might be showing a p value below the set cut off of 0.05 or 0.01 (whichever is applicable here). However, this does not ofcourse always translate as actual practical difference. The fact that they had to use those terms makes me think that the difference is only statistical and maybe not important clinically.

5) I am not sure if causation has been conclusively proven. There were no rules regarding how the money was to be used- there was no blinding and people who received the money knew they were part of a study looking at all these outcomes. That itself could have influenced the results obtained. Further details about the intervention and control groups would be required to see if they were comparable. I think the investigators were also eager to state the improvements noted in the intervention group- even though it might not be enough to prove causation, it is a significant study.

6) I think the mechanism mentioned here is regarding getting full-time jobs. People not working would have been able to get transport to go for interviews. People having part time jobs would be able to take time off for interviews. Although quite impressive, I dont think the money is the only thing here. People also need to be motivated to go for interviews- and I think just the awareness of being in such a study would have stimulated them to apply for interviews and use the money in a productive way.

Fathima

Thank you for sharing this interesting article Sujit. I think universal basic income is a promising humane idea and does need further study. This study comes from these people https://www.stocktondemonstration.org/ but there is no academic paper that I can find associated with it. I imagine that they are writing one and it will be published in due course.
Thanks also for Colette, Margaret and Fathima for your thoughts which I have found helpful.
1. Randomised controlled non-blinded trial
2. PICO - means patient population, intervention, comparison with controls, outcome
Patient population - people living under the median income in Stockton, California
Intervention supplement of $500 a month for 2 years
Controls no extra money
Outcome - in a scientific paper we would told what the primary and secondary outcomes were. In the Guardian article we are just told about positive findings. These were higher rates of employement, less debt and better emotional health
3. 28% and 32% - the rates of having a full time job at the start of the study. These are very low. Stockton was chosen for this study as it is a very deprived area. I am struggling to find US data on employment rates -but it looks like employment is usually at 90% and 85% of people work full time which would mean that we would normally expect rates of about 77%.
4. https://www.stocktondemonstration.org/ tells us that SF-36 a mental health questionnaire was used to rate mental health, but we are not given any numbers to compare the scores on this measure for cases or controls.
5. We do not have the necessary data to answer this question. In any case, we would need more data from other studies and consideration of the Bradford-Hill criteria to begin to suggest causation.
6.The researchers said that the extra $500 per month was enough for people with part-time jobs to take time off so they could interview for full-time jobs that offered better pay. They also said the money could have helped people who weren’t working at all find jobs by allowing them to pay for transportation to interviews

Judith

Well done everyone. If you keep this up, I'm going to have to start writing more difficult questions!

I assumed this was a randomized study, but the imbalance between arms on the baseline employment level makes me wonder if Fathima is right, that this was a quasi-experimental study. Or perhaps this was a small RCT, in which case imbalance is to be expected.

As Judith found, the study's findings have been published in the newspaper, before peer review. The average newspaper reader may not be able to come up with the limitations you have, and end up with expectations about universal basic income which aren't grounded in reality.