Tags: children's mental health, program evaluation, SAMHSA, system of care
For two years, I worked full time in the field of children’s mental health program evaluation and research as a “Research Project Coordinator.” My boss, the “Principal Investigator,” and I had been contracted to implement a standardized, predetermined program evaluation of what is called a “system of care” — a program that coordinates and delivers services to children with mental health challenges. The system of care had been set up with money from SAMHSA [Substance Abuse and Mental Health Services Administration], part of the Department of Health and Human Services.
Any community that received funds from SAMHSA to start one of these “systems of care” – and there were over 144 of them! – had to agree to implement the full data collection protocol, which required in-depth multiple choice interviews be administered to about 250 families. Researchers were also expected to access extensive school records, court records, and medical records from the community mental health center. Again, these research methods were implemented over and over, in every one of those 144 funded communities.
In short, the “Children’s Mental Health Initiative” (as SAMHSA calls it) is the largest standardized children’s mental health research project ever conducted. The datasets collected by this massive undertaking – nearly 20 years’ worth of data on tens of thousands of families – are owned by the government and managed by a contracted firm called ICF International, which, oddly enough, also has extensive contracts with the Departments of Defense and Homeland Security. Go figure.
So that was the project, and I was its local coordinator. I worked extensively with the regulatory agencies that protect human research subjects’ rights to get their approval for our proposed “research methods.” I called potential participants and, following the script, assured them that being in the study was a chance for “their voice to be heard” and to “ensure that other children with challenges like their child’s could get services, too.” I conducted interviews with mothers, grandmothers, uncles, and the children themselves (when they were old enough to be in the study). I attended children’s mental health research conferences where we presented our data and listened to other communities present theirs’. I attended numerous meetings in our designated area of evaluation (the community, we always called it), listening to social workers and other helping professionals talk endless circles around the project, its goals, its implementation, and something they called sustainability.
Sustainability inspired complicated emotions in these folks. Each and every person employed by the system of care would lose his job in 5 years (that’s when the federal funding ran out), if the specter of sustainability was not made more substantial.
Now, a federally funded “system of care” is a real gravy train. A community mental health center thus financed can easily employ twice as many, perhaps even three times as many, social workers and “helping professionals” as before. Some people were literally paid to do nothing more than exist and lend the program more credibility by their impressive titles; the “Technical Assistance Coordinator” and “Social Marketing Associate” come to mind. And then there were the fantastic perks: free trips to “conferences” at lovely resort destinations (how’s Tampa in late February/early March sound?), fancy business lunches provided on a near weekly basis (they call it “community engagement”), and, best of all, business cards that say you’re making a difference in children’s lives – a real boost to the self-esteem, which is priceless, isn’t it?
Most of the staff seemed to know that they couldn’t keep riding the gravy train forever, but the burning question – would they even have jobs in 5 years? – had to be answered. So sustainability was a real presence in those meetings, a creditor who couldn’t be evaded forever, or a god that must be appeased.
But what’s the one thing that nobody – not even the gods – can question? SCIENCE. It’s objective. It’s proven. It’s Fact. Science as published in peer-reviewed, “scholarly” articles, as glossily summarized in abstracts and press releases, as cited by thousands who have never read the research in its entirety. If you choose to discount it, then you’re an irrational being, not worthy of consideration.
So science, in the form of program evaluation, was the answer to the sustainability question. The system of care needed research that “proved” it was “effective,” that justified its continued existence, and that generated visually stimulating graphs with lines that went up (it really was that simple). With their irrefutable SCIENCE in hand, they would apply for new grants, secure renewed funding, and keep the gravy train chugging along. This was not just the local solution to the sustainability problem – it was national and it was built right into the Children’s Mental Health Initiative by SAMHSA from the very beginning.
This, by the way, is a classic example of a synthetic Hegelian dialectic put in place by an institution to achieve its pre-determined goals with a semblance of grassroots mobilization; otherwise known as the “problem-reaction-solution” paradigm. In this case, it went something like this:
- Problem: in 5 years you won’t have a job
- Reaction: I can’t miss a mortgage payment! What do I do?
- Solution: justify the continued existence of your job by participating in and encouraging a truly massive data collection initiative, the depths and true purpose of which you know nothing about, an undertaking that under different circumstances you might question critically…
It hits home at a very personal level. The threat of losing your job is literally a threat to your survival (and in the case of most of these employees, a threat to the survival of their children). Animal — and maternal — instincts are activated, and critical thinking is no longer a part of the equation.
A critical thinker would question the protocol that SAMHSA gave us. Is it appropriate to ask a parent to “list 5 things you don’t like about your child,” or “True or False: I turned out to be a worse parent than I thought,” (as we did in every interview)? Is this “data” of any scientific value whatsoever? Why must the protocol as a whole be so incredibly negative, pathologizing, and stigmatizing? What is all this data really for, and what is the true purpose of the data collection exercise?
Such questions were taboo. If someone insisted on asking them anyway, they were met with simple, pat answers, such as “it [the protocol] is not perfect, but it will help us help kids.”
We were doing it in the name of job security. We were doing it to survive. Problem-Reaction-Solution.
In countless meetings, workshops, and teleconferences, our mission as evaluators and “research scientists” was made clear to us: we needed to prove that the system of care was effective to ensure more funding, on both the local and national levels.
Obviously, with this being the setup, bias in our research was maximized from the very beginning. In the script I read to potential participants in the study (which was “officially approved” by the Human Subjects Institutional Review Board), I was instructed to say: “We’re hoping to prove that the system of care works. That way, similar programs can be funded in the future.” From the first moment of contact, participants were notified what outcomes we, the researchers, expected.
Amazingly, many families would agree to the first interview. Few agreed to a second, however (there were supposed to be 5 per family; one every 6 months). They knew something was up.
Nevertheless, the data collection project went forward more or less as planned – continues on nationally to this day.
And so I return, now, to those questions I wasn’t allowed to ask at the time: What was all that data really for?
Sometimes we said it was to prove that the system of care improved outcomes for children and families involved in it, but that’s not the answer. Wouldn’t we have stopped to ask the families what “improvement” meant to them if that were the nature of our quest?
Other times, when we professionals were talking amongst ourselves, we said it was to secure more funding; but that could easily have been achieved with far less data then we collected during the burdensome, 2 ½ hour long interviews. The annual report to Congress, the supposed culmination of our efforts, was usually only 15-20 pages long, with only very small portions of the data we collected represented in its primary colored pie graphs.
Why did SAMHSA need to win our compliance with this effort so clearly deserving of critique?
What is the true purpose of the data collection enterprise?
Tags: Einstein, ignorance, mediocrity, mercenaries
If we knew what we were doing, it wouldn’t be called Research.
this quote being the only decoration which hangs in the office of my boss, the Research Scientist
I remember the day I first crossed the threshold of his office – for my job interview. That quote, framed and prominently displayed in an otherwise bare office, gave me pause. “Is that supposed to be cute?” I thought.
I wondered if Albert Einstein ever even said that. And if he did, does it somehow make more sense — and sound more like something a believer in research/science would say — in German? Or in context?
What kind of a researcher would make this his motto, even as a joke?
But then I turned my attention to the man in front of me. Destitute and desperate for a job at the time, I did my best to put these questions aside and form a good opinion of the Research Scientist who had the power to radically change my position and my general outlook.
I really wanted that job.
2 years later, after gaining a sickening familiarity with his research “methods,” I’ve got a much better handle on what that quote means to him. I think he takes a great deal of comfort in it. If Albert Einstein, the standard cultural trope for “genius” and “scientist,” thought this – then maybe he cut corners, only examined data that was convenient (writing off the rest as “faulty” or whatever), and just plain made shit up sometimes, too! By pulling Einstein down to a comfortable level of mediocrity, this Research Scientist is suddenly in much better company. It’s not quite “I’m just as good as Einstein!” – it’s more like “Einstein was just as bad as I am!”
PART II to our conversation
Our second conversation on the childhood bipolar “fact” sheet took place on Friday. And I spent the weekend pondering. You see, I realized that he probably never read the sheet all the way through. I inferred this from his lack of knowledge of what it said beyond the first two paragraphs – which I had read to him over the phone — and his mistaken attribution of the sheet to NAMI.
He didn’t read it to protect his carefully maintained ignorance/innocence. Basically, if he didn’t know what it said, he wouldn’t be convinced that ethically, morally, something needed to be done. And he would therefore maintain his innocence. [I told you I’m familiar with his methods!]
What to do? How to provoke some actual critical thinking about this? He likes to think with his self-preservation gut (prime directive: preserve ignorance/innocence). How to get beyond those very formidable defenses against data and reach the actual Research Scientist with conscience?
I thought I’d give it one more shot.
This time, I went with email. It would appear less confrontational, and maybe not trigger those strong defensive responses.
***From: ALT Sent: Monday, April 04, 2011 8:19 AM To: Research Scientist Attachments: bipolar info sheet.pdf
See attached to learn my objections to the childhood bipolar info sheet. I welcome your comments or questions. I hope you will give this information all the attention an “issue close to your heart” deserves.
I think your use on Friday of the phrase “company line” was very appropriate. These inaccuracies certainly promote a company line – lifelong, chronic illness never to be cured but only to be suppressed with an ever‐changing cocktail of expensive, patented drugs. It is a very convenient narrative for a pharmaceutical company.
But we are not a company. We are “researchers,” are we not?
His response (in about a minute!):
You cannot understand how close this issue is to me. SO forgive me if I do not meet your expectations related to this.
What I’d like to say to this Research Scientist (if the conversation weren’t so clearly over) is that it’s not about “my expectations.” It’s not about me at all. It’s about doing what’s right for the families and youth who will be affected by this information.
March on, Soldier. But what are you fighting for?
Did you know that in police or heavily militarized states (Mexico, as one of many examples), where soldiers and policeman treat much of the country as an occupied territory, the citizens and populace as enemies – a policeman or soldier is never stationed in his hometown, or even home region? They are stationed as far away from home as possible.
Because there are some things people typically just won’t do to their brothers, relatives, childhood companions. Point a gun in their faces. Dehumanize them. Treat them as fundamentally “other.” Kill, rape, terrorize.
But strangers? Folks that look different? And in the case of Mexico, speak different languages? Come from radically different cultural heritages?
Yeah. You can train soldiers to do it to them. Incrementally, step-by-step. It can be done.
I shared this information, long ago, with my boss the Research Scientist. And then I mentioned the fact that nearly all of his research projects take place far from where he lives. One is a 3 hours’ drive away; another is 18 hours. He’s often told me that observation troubles him.
Why is it so easy to think of him as a mercenary?
When it comes to childhood bipolar disorder, whose “company line” is he soldiering, guarding with his silence? He styles himself as a children’s mental health advocate, but what is he really fighting for?
Tags: childhood bipolar disorder, program evaluation, research scientist
The plaque on the door of my boss’s office says “Research Scientist.” He’s a professor at a major university, a specialist in the intersection of children’s mental health and schools (special education, specifically). He’s also a national expert on a specific type of children’s mental health service system, known as a “system of care.” This is what he did his dissertation on, and he is regularly consulted by governmental and corporate policymakers because of his “expertise” in this area and his access to huge data collections.
Right now his biggest grant (and therefore biggest research project) is with a system of care in our state. They have contracted him to help them improve their practices by making them more “evidence-based” – ie informed by data and statistics. He and his team (including me) are responsible for gathering data about the families in the community in need of mental health services, and then formulating that data into policy recommendations for the system of care. In general, we, the evaluation team, are charged with ensuring that the community and the system of care in particular have the best evidence possible to inform their service design.
About a week ago, a “parents of seriously mentally ill children” support group from the area that this system of care serves sent out a newsletter, their first. The newsletter contained in it two pages of “information” about [childhood] bipolar disorder. Unfortunately, these two pages were riddled with inaccuracies and outright lies of a – frankly – disturbing nature. I have outlined the bulk of those inaccuracies here.
These are a few particularly striking quotes:
Several factors may contribute to bipolar disorder including… abnormal brain structure and brain function.
(from the Bipolar Disorder fact sheet)
Right now, there is no cure for bipolar disorder. Doctors often treat children who have the illness in a similar way they treat adults. Treatment can help control symptoms. Treatment works best when it is ongoing, instead of on and off.
(from the Bipolar Disorder fact sheet)
The day that I received this newsletter, I brought it to my boss’s (remember – Research Scientist) attention. I read him the first section of the info sheet. His response was to groan and say “Yeah… that sounds bad.” I asked him what we were going to do about it.
He paused. He stalled. He mumbled. He said things like “this issue is very close to my heart. You know that.”
And then he said, “Let me talk to my mentors about it. I’ll get back to you on Monday.”
He’s been avoiding me ever since. I mean, just straight not talking to or communicating with me in any way.
So I bided my time. And finally, this morning, he called to catch up on what’s happened this week, gleefully prattling about minor business items. Pretending the whole conversation about that info sheet had never happened. Finally, he said, “Well, that’s all there is on my agenda. Anything else you can think of?”
YES. As a matter of fact there was something else I could think of – the elephant in the room. The childhood bipolar disorder “fact” sheet. The fact that we are supposed to provide good data to the community, and this sheet contains almost nothing but bad data.
This is how our conversation went:
Conversation with a Research Scientist
ALT: So what do you think about the childhood bipolar disorder info sheet?
Research Scientist: The what?
ALT: The childhood bipolar disorder sheet that the parent support group sent around.
Research Scientist: Well, I asked my respected friends and colleagues, I talked to a few of them… and none of them seemed disturbed by it like you were. Archaic, yes. But that’s just the status quo. As one of my friends said, “that’s the company line.”
ALT: What company?
Research Scientist: Oh… [backpedaling]… oh, he meant the US Government, I think, not a company.
ALT: I don’t really care what your colleagues think about it. What do you think about it?
Research Scientist: Nothing.
ALT: (laughing) What?
Research Scientist: I don’t think anything about it.
ALT: That’s impossible. You can’t read it and not think anything about it.
Research Scientist: I don’t think anything about it. [pause] I think I support the work of the parent support group. I support what they do.
ALT: That’s great… but what do you think about what’s written on the sheet about childhood bipolar disorder? Do you think it’s accurate?
Research Scientist: What do you want me to do?
ALT: YOU have to make a decision. I haven’t been in the adult world very long, but that’s part of it, isn’t it? Something comes along, and it’s inaccurate, and it’s your responsibility to address that! Do you think this info sheet is accurate?
Research Scientist: Well, no it’s not accurate. But that’s the status quo.
ALT: So you DO think something about it. You think it’s inaccurate.
Research Scientist: ALT, [laughing] I read things that are inaccurate every day without doing anything about them. In the New York Times, in the Huffington Post.
ALT: That’s different; those are newspapers.
Research Scientist: No – this is a newsletter so it’s basically the same.
ALT: Those papers are read by millions of Americans across the country. This newsletter is distributed by the parent support group to the parents in our area. The area where we were contracted to help them improve their practice by making it more evidence-based. This is bad data. And you know it.
Research Scientist: So what do you want me to do? Tell me what you want me to do.
ALT: I want you to tell me your thoughts about this info sheet, and then I want you to make a decision about what to do.
Research Scientist: Ok. [… long pause…] Well I have decided to ask Sharon [member of parent group that put the newsletter together] about it.
ALT: Do you have any idea what question you might ask Sharon?
Research Scientist: No. I haven’t decided that yet. And I have a few weeks before I see her, so I have time to think about it.
ALT: Ok. Well, I look forward to that conversation.
[… extended silence…]
I hardly need add that my boss, the Research Scientist, will not ask Sharon anything. By delaying action needlessly (why not email or call Sharon today instead of waiting three weeks?) he hopes that I will get the message to drop it and he will be allowed to nothing. And to preserve a semblance of ignorance (to him, that equals “innocence”) on the topic.
Some other things: note how this Research Scientist will do anything to avoid stating his own opinion about the information contained on the sheet. He will quote other’s opinions, but even with all my direct questioning only fails to evade me once. He does not find the accuracy of the information sheet particularly relevant, it seems. Likewise, even though he is the principal investigator (he has ultimate responsibility in this grant), he will do ANYTHING to avoid making a decision. Even if that means asking an inferior, young and inexperienced minion with only a BA in an unrelated field what to do. [I am speaking in terms of the academic hierarchy in place, which is rigid and unforgiving. I am lower than the lowliest pot-smoking grad student here and my opinions count for basically nothing], THIS RESEARCH SCIENTIST CANNOT MAKE DECISIONS FOR HIMSELF. Lost the power to do it years ago. Another result of fine academic training, I expect.
Also, I think his use of the phrase “company line” is incredibly telling.
Folks, this is what the research scientists look like in real life. BEWARE.
[Part II of the conversation here]