SPECIAL ARTICLE Evaluating the RSBY: Lessons from an Experimental Information Campaign
Jishnu Das, Jessica Leino
Launched in 2008, the Rashtriya Swasthya Bima Yojana provides financial protection from health shocks for poor households. This paper discusses findings from an experimental information and education campaign and household survey carried out in the first year of the programme in Delhi. First, the IEC had no impact on enrolment, but households who were part of the household survey sample and therefore received information closer to the enrolment period were 60% more likely to enrol. Second, there is little evidence that the insurance company selectively enrolled healthier households. Instead, hospital claims were lower for households who received the IEC and for households who received both the survey and the IEC, suggesting that the marginal household enrolled was in fact healthier. Implications for the programme and its evaluation are discussed in the light of these findings.
We thank Robert Palacios for helpful comments and Risha Hess, Anvesha Khandelwal and Seema Grover for assistance in the IEC, household survey and data management. The findings, interpretations and conclusions expressed in this paper are those of the authors and do not necessarily represent the views of the World Bank, its executive directors, or the governments they represent.
Jishnu Das (jdas1@worldbank.org) and Jessica Leino (jleino@worldbank.org) are with the World Bank.
1 Introduction
I
Using the experimental nature of the IEC and the equalprobability sampling in the household survey, we estimate causal e ffects of the IEC on enrolment and hospital claims using administrative programme data. With the IEC and the household s urvey, there are four “blocks” of households in our data (IEC only, IEC + household survey, household survey only, neither); the causal impacts of any treatment block can be computed by means-comparisons across blocks.
We report three results. First, the IEC by itself had no impact on enrolment. This could be because the IEC was carried out two months prior to the effective start of a delayed enrolment process, and because not all administrative details of the enrolment process were available at the time of the IEC. It is also possible that, because rumours had spread that if people did not enrol in RSBY they would not be able to obtain supplies through the public distribution system (PDS), the IEC actually discouraged enrolment by dispelling these fears. Second, and contrary to our expectations, the household survey itself increased enrolment. The household survey, which (coincidentally) took place immediately prior to and during the enrolment period, asked questions about the scheme and households’ knowledge of the scheme as well as households’ health status. It is likely that asking these questions (as well information that surveyors may have shared on the scheme as part of the interview process) increased the salience of the scheme among surveyed households and altered the likelihood of enrolment. Behaviour change as a consequence of observation, termed a H awthorne effect, has been noted in several r ecent studies
Economic & Political Weekly
EPW
SAMEEKSHA TRUST BOOKS
China after 1978: Craters on the Moon
The breathtakingly rapid economic growth in China since 1978 has attracted world-wide attention. But the condition of more than 350 million workers is abysmal, especially that of the migrants among them. Why do the migrants put up with so much hardship in the urban factories? Has post-reform China forsaken the earlier goal of “socialist equality”? What has been the contribution of rural industries to regional development, alleviation of poverty and spatial inequality, and in relieving the grim employment situation? How has the meltdown in the global economy in the second half of 2008 affected the domestic economy? What of the current leadership’s call for a “harmonious society”? Does it signal an important “course correction”?
A collection of essays from the Economic & Political Weekly seeks to find tentative answers to these questions, and more.
Pp viii + 318 ISBN 978-81-250-3953-2 2010 Rs 350
Windows of Opportunity
By K S KRISHNASWAMY
A ruminative memoir by one who saw much happen, and not happen, at a time when everything seemed possible and promising in India. K S Krishnaswamy was a leading light in the Reserve Bank of India and the Planning Commission between the 1950s and 1970s. He offers a ringside view of the pulls and pressures within the administration and outside it, the hopes that sustained a majority in the bureaucracy and the lasting ties he formed with the many he came in contact with. Even more relevant is what he has to say about political agendas eroding the Reserve Bank’s autonomy and degrading the numerous democratic institutions since the late 1960s.
Pp xii + 190 ISBN 978-81-250-3964-8 2010 Rs 440
Global Economic & Financial Crisis
In this volume economists and policymakers from across the world address a number of aspects of the global economic crisis. One set of articles discusses the structural causes of the financial crisis. A second focuses on banking and offers solutions for the future. A third examines the role of the US dollar in the unfolding of the crisis. A fourth area of study is the impact on global income distribution. A fifth set of essays takes a long-term view of policy choices confronting the governments of the world. A separate section assesses the downturn in India, the state of the domestic financial sector, the impact on the informal economy and the reforms necessary to prevent another crisis.
This is a collection of essays on a number of aspects of the global economic and financial crisis that were first published in the Economic & Political Weekly in 2009.
Pp viii + 368 ISBN 978-81-250-3699-9 2009 Rs 350
1857
A compilation of essays that were first published in the EPW in a special issue in May 2007. Held together with an introduction by Sekhar Bandyopadhyay, the essays – that range in theme and subject from historiography and military engagements, to the dalit viranganas idealised in traditional songs and the “unconventional protagonists” in mutiny novels – converge on one common goal: to enrich the existing national debates on the 1857 Uprising.
The volume has 18 essays by well-known historians who include Biswamoy Pati, Dipesh Chakrabarty, Peter Robb and Michael Fisher. The articles are grouped under five sections: ‘Then and Now’, ‘Sepoys and Soldiers’, ‘The Margins’, ‘Fictional Representations’ and ‘The Arts and 1857’.
Pp viii + 364 ISBN 978-0-00106-485-0 2008 Rs 295
Available from
Orient Blackswan Pvt Ltd
www.orientblackswan.com Mumbai Chennai New Delhi Kolkata Bangalore Bhubaneshwar Ernakulam Guwahati Jaipur Lucknow Patna Chandigarh Hyderabad Contact: info@orientblackswan.com
august 6, 2011 vol xlvi no 32
d iscussed in Zwane et al (2011). Third, households chosen for the IEC and IEC/household survey ultimately reported lower claim rates, suggesting that the marginal household enrolled as a result of this information provision was healthier than the average household. Insurance company profits therefore were higher for the set of households exposed to the IEC and household survey. This result is consistent with the incentives built into the design of the RSBY – the programme explicitly covers pre-existing illnesses in the belief that the “first enrollees” would be “sicker” households. Consequently, the marginal households that joined up as overall enrolments increase should be progressively healthier with lower claims.
These results raise the substantive issue of the impact of information and exposure to the programme on enrolments and hospital use. Information campaigns are likely to be extremely time sensitive, and need to be carefully timed and managed to coincide with actual enrolment dates. The marginal take-up as a result of even well-timed campaigns may be low, as household enrolment decisions depend on many factors outside the influence of the IEC. Furthermore, households who received both the IEC and the household survey, several months apart, reported the highest e nrolments. This suggests that multiple doses of information may have a reinforcing effect.
Was this additional enrolment good for the insurance companies? If we disregard the cost of the campaign, almost certainly: there was an additional net profit of Rs 66 on every household enrolled under the IEC and household survey. It seems likely that in urban areas, a well done IEC could increase insurance company profits. What a well done IEC would look like and how much it would cost remains an open question.
These results also highlight methodological issues of how to evaluate the impact of the RSBY programme. One approach used in the literature combines a baseline household survey with an experimental encouragement design, where the encouragement could take the form of information or subsidies (King et al 2009; Thornton et al 2010). To the extent that the encouragement i ncreases enrolment in the programme and is experimentally a llocated, intention-to-treat (ITT) estimators and instrumental variables can be used to back-out the impact of the programme (under certain assumptions). But our programme administrative data shows that the baseline household survey itself provides i nformation to households and increases enrolment. This raises the issue that without such independently available sources of administrative data, it would have been difficult to discern the effects of the survey on enrolment and programme outcomes since there is no information available on households who would not have been administered a survey. In Section 5, we discuss the options available for evaluating the broader impact of the RSBY programme when surveys alter participation decisions.
The remainder of our paper is as follows. Section 2 describes the RSBY and its implementation in Delhi. Section 3 describes our data and empirical strategy. Section 4 documents the results and Section 5 discusses broader methodological implications and concludes.
2 RSBY: A Background
India’s health system was planned largely as a three-tiered referral system of primary healthcare centres, district hospitals and specialised hospitals where care would be sought primarily
Economic & Political Weekly
EPW
through the public sector and financing would be based entirely on a supply-side model. But by the 1990s, studies showed that
To redress these problems, the government launched its flagship health insurance product for the poor in 2008. The RSBY o ffers a micro-insurance product for households designated as “below the poverty line” (BPL) and aims to cover up to 60 million households throughout the country over the next five years. The objectives of the RSBY are to provide financial protection for households affected by major health shocks and improve health outcomes for poor households. The RSBY is a voucher-like intervention; by combining consumer choice among competing facilities with a demand-side subsidy, it aims to provide greater financial protection for poor households and foster better quality care through increased competition. There are several unique features of the programme.1
First, insurance companies are selected by competitive bidding in each district and receive a premium for every household e nrolled by them in the scheme (typically paid 75% by the central government, 25% by the state government).
Second, insurance companies empanel in-patient care facilities (ICFs); they then reimburse ICFs for in-patient care provided to enrolled households. ICFs may be either public or private; public facilities may retain payments from the RSBY in self-governed societies known as Rogi Kalyan Samitis.
Third, eligible households (those identified as BPL by their states) can enrol in the programme by paying Rs 30, in return for which they receive a smart card. A maximum of five members may enrol from any family, including the head, spouse and up to three dependents of the head of the household. The smart card permits any registered member of the household to visit any ICF for in-patient care for any of the (approximately) 700 surgical or medical procedures. The prices for each procedure are fixed by the state, and an enrolled household is covered for a maximum of Rs 30,000 of in-patient care each year. Transactions are cashless for households and require only the fingerprint-verification of the user to deduct the cost of the procedure from the remaining smart card balance.
As of January 2011, the RSBY has been rolled out in over 400 districts in 26 states and in the first two years has covered more than 18 million households with around 55 million individuals. The smartcard data system that automates and centralises data collection p rovides administrative data on enrolled households and procedures performed at ICFs on a centrally maintained server.
SPECIAL ARTICLE
During the initial rollout of the programme, several important shortcomings were identified. First, insurance companies are paid for every household enrolled, not for every individual enrolled. Therefore, they have incentive to enrol the first member of a household and zero incentive to enrol any other member. A lthough it is difficult for insurance companies to deny enrolment, one can imagine that they do not actively promote enrolment among additional household members; they may even assure households that enrolling a single member is sufficient. We see some evidence of this in the numbers. Across the programme, the average number of members enrolled per household in the RSBY at 2.6 is much lower than the average number of eligible household members of 4.6.2 Second, there is currently no information on what happens at hospitals or how patients are diagnosed and assigned treatment codes. Therefore, we cannot ascertain whether hospitalisation was medically necessary, and if so, what the impact of this hospitalisation was on the health of the household. In an analysis of another health insurance scheme targeted to the poor, Desai (2009) points out that 43% of the g ynaecological claims were for hysterectomies that were actively encouraged by participating hospitals. Within the RSBY, presently insurance companies are focused primarily on controlling outright fraud rather than examining medical necessity.
Implementation Issues in Delhi
Delhi was the first state to start enrolling households in the RSBY programme. Understanding how the programme functioned in Delhi highlights the inherent uncertainties of programme implementation and that these findings may not generalise to other contexts.3 We emphasise three characteristics of the process in Delhi. These are (1) the list of eligible households and the state’s own IEC; (2) the pricing of services; and (3) the choice of the i nsurance company.
First, the nature of urban poverty and technical implementation issues limited the state’s ability to provide clear information and generated considerable uncertainty about the purpose of the scheme and the process of enrolment. The initial list of beneficiaries in Delhi’s programme included 1,47,000 households designated as Antyodaya households, considered the poorest BPL households in the state. Early on, we discussed with the implementing officer in the state’s nodal agency the difficulties of IECs in an urban setting with many eligible households in slums and very low-income neighbourhoods scattered throughout the city. We were told that the state would advertise in newspapers and the night before enrolment began, the circle in-charge (Delhi has 72 circles for purposes of the food distribution under the PDS) would be responsible for contacting each of the eligible households and directing them to enrol in the programme. Since some circles had more than 8,000 households, it was difficult for us to see how the planned awareness activities would work in practice. As it turned out, Delhi reported the lowest enrolment of all participating states in the first year.4 In general, information about the programme in Delhi was sparse and rumours circulated widely about the actual purpose of the RSBY card.5
In addition to problems with notifying potential beneficiaries, technical issues with the smart card delayed the starting enrolment date by three months. Once enrolment started, the field key officer (FKO) responsible for verifying the identity of beneficiaries prior to enrolment became a bottleneck. The FKOs – who were all government employees – often did not come to work, would take long breaks, and would arbitrarily turn people away. Given the frequency of such problems, households had to make a real effort to enrol in the RSBY. In some cases, households would have to make three to four visits to the enrolment station before they found the FKO and the FKO decided that their papers were in
o rder for enrolment.
Second, the choice of hospitals and the pricing scheme limited the incentives for ICFs to actively seek out patients from the RSBY scheme for treatment. Delhi decided not to empanel public facilities on the grounds that public hospitals were supposed to provide care for the poor free of cost and by participating in the programme, incentives would be skewed towards providing care for RSBY beneficiaries – because the reimbursement from RSBY procedures would be retained by the hospital. Furthermore, the government decided not to alter the price list that the centre had made available as a reference list based on prices in smaller towns. As the programme proceeded, it became clear that these prices were too low for many procedures and that in some cases hospitals were turning away patients.6 These factors could affect the interpretation of hospital use statistics that we report here.
Finally, Oriental Insurance Company, a parastatal with potentially wider objectives beyond profit maximisation, won the bid for Delhi. In an analysis of 24 districts, those with private insurance companies reported significantly higher enrolments.7 This suggests that parastatal insurance companies (a) operated under lower incentives; (b) operated in districts where enrolments were likely to be lower; or (c) were less likely to exert pressure on the state on important issues (such as FKO presence) that could i ncrease enrolment. While the reality is almost certainly a combination of these factors, during the first year enrolment process it appeared that Oriental was not able to collaborate effectively with the Delhi state government in terms of choosing enrolment sites, improving the IEC, and ensuring FKO presence.
Due to these unique implementation circumstances, the results of this study apply only to Delhi in the first year of enrolment. The main lesson we would thus submit is not necessarily the s pecific programme impacts, but the experience from this pilot for designing evaluations of the programme in different states and in understanding how (and whether) IECs could add value to the functioning of the programme.
3 Data and Empirical Strategy
This paper relies upon data from several sources that was collected over a one and a half year period. The sample frame list provided by the Delhi government in early 2008 contained information on 23,836 eligible households in the six administrative circles selected for the IEC.8 Of all households in the sample frame, 7,500 were then randomly sampled for the IEC and 3,000 (1,500 that received the IEC and 1,500 that did not receive the IEC) were randomly sampled for the household survey. There are thus four “blocks” of households (IEC only, IEC + household survey, household survey only, neither) in our data; because of the random
august 6, 2011 vol xlvi no 32
allocation of households to blocks, causal impacts of being in a given block can be computed simply by comparing the means across blocks. The block sizes are not the same; there are 6,004 households in IEC only block, 1,496 in the IEC + household survey, 1,484 in the household survey only and 14,852 in the group that received neither. All the post-intervention data are based on administrative records of enrolment and hospital use.
The timeline of the study was altered several times. Enrolment was originally slated to begin in March 2008, with the IEC to be carried out in January and February 2008. However, enrolment officially began in April, with the early months plagued by implementation problems. Enrolment picked up pace only around June/July 2008. The home visits component of the IEC took place in late April and early May 2008. Our household survey ultimately ran from June to September 2008. The administrative data on enrolled households as well as on all insurance transactions completed by enrolled households was obtained at the conclusion of the policy period in April 2009.
The IEC was conducted by an organisation with substantial e xperience implementing community-based interventions in the areas selected for the intervention. The IEC employed a “surround and engage” strategy so that the target audience would hear about the RSBY scheme through multiple channels. A letter was mailed to each household in the treatment sample as an i ntroduction and this was followed with a home visit by a trained field officer. A leaflet with easy-to-read text as well as pictorial cues for low literacy audiences that outlined the key benefits of RSBY, the procedure for enrolment and a helpline number was also distributed during the household visit. As part of the IEC, posters also were put up at ration shops and at sites such as the local flour mill where members of the target audience were likely to congregate. The organisation found that 27% of households could not be reached for household visits, the vast majority due to the poor quality of the sample frame list.
To assess the characteristics of households enrolled in the programme with and without the IEC, the data collected from the household survey sample contains information on self-reported health status and questions on activities of daily living for all household members. It also has questions on household expenditures on health and details about any health shocks faced by the household in the past year. The household survey also contains administrative data generated through the programme also has limits. Since there is no information on non-enrolled households in the administrative data, we cannot compare the characteristics of households who were not surveyed to those who were.
The administrative data on enrolled households and hospital use were matched back to the sample frame and to household survey data for the subsample of households that completed the household survey. The empirical strategy relies on the ITT strategy as some households targeted for the information campaign could not be reached and ultimately 2,980 of the 3,000 sampled households completed the household survey.
4 Results
The analysis that follows examines the effect of both the information campaign and the household survey by classifying all households that were targeted to receive the IEC/survey as treated households.
4.1 Enrolment
The IEC had little impact on enrolment, except in conjunction with the household survey. Figure 1: Enrolment and Utilisation Rates across
Groups of Households
Figure 1 shows mean
4
enrolment and utilisation rates for the four 3 types of households in our study. Enrolment 2 rates were 24%-25%
1
among households that did not receive the
0
Proportion enrolled Proportion utilised Control IEC HH Survey IEC+HH Survey
household survey, and
The figure shows the fraction of households who enrolled in increased to 38%-42% the RSBY scheme and the fraction who reported a hospital claim within the RSBY scheme from administrative data for four
for households that
groups of households: the control group that received neither an IEC nor a household survey; the group that
received the house
received only the IEC, the group that received only the hold survey. The con-household survey and the group that received both. Enrolments were higher for the group that received the trol households (those household survey and those that received both the household survey and the IEC, but not for those who
without the IEC or
received only the IEC. Claims were lower in the groups that household survey) in received the IEC. Delhi had enrolment rates of 24%. The results reported in Figure 1 are unconditional means; as the distribution of households in each group varied slightly across administrative circles and enrolment rates varied substantially across circles (from 12% to 87% of households), Table 1 reports
information on attitudes towards risk, Table 1: Impact of IEC and Household Survey on Enrolment, Utilisation and Profits
household assets, income and borrowing, and demographic characteristics.
Finally, administrative data collected automatically through the RSBY smart card technology and the programme management information system (MIS) are used to assess the impact of the IEC and the house-
Dependent Variable:
HH sampled for IEC
HH sampled for survey
| Percentage Enrolled | Number Enrolled | Percentage | Total Claims (Rs) | Net Profits Per | ||
|---|---|---|---|---|---|---|
| (Conditional on | Utilised Insurance | (Conditional | Policy (Rs) | |||
| Enrolment) | (Conditional on | Upon | (Conditional | |||
| Enrolment) | Transaction) | Upon Enrolment) | ||||
| ols | probit | ols | ols | probit | ols | ols |
| (1) | (2) | (3) | (4) | (5) | (6) | (7) |
| -0.90 | -1.03 | 0.02 | -1.46** | -1.30*** | 246** | 74** |
| (0.68) | (0.73) | (0.05) | (0.48) | (0.31) | (106) | (25) |
| 7.99*** | 9.26*** | 0.03 | 0.52 | 0.57 | -484 | 0.6 |
| (1.76) | (2.22) | (0.08) | (1.42) | (1.30) | (1222) | (75) |
| hold survey on enrolment and claims rates. | HH sampled for both | 4.83** | 5.02*** | 0.06 | -0.66 | -0.60 | -719 | 53 |
|---|---|---|---|---|---|---|---|---|
| Note that without administrative data, esti- | IEC and survey | (1.27) | (1.10) | (0.08) | (1.70) | (1.49) | (1018) | (104) |
| mating the effect of the household survey on | Constant | 19.63*** (0.39) | 25.20 | 3.45*** (0.02) | 4.98*** (0.34) | 45.08 | 4945*** (262) | 298***(14) |
| enrolments and transactions requires both a | Circle FE | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| follow-up household survey and an extension | Observations | 23836 | 23836 | 6407 | 6407 | 6407 | 351 | 6407 |
All regressions include administrative circle fixed effects. Marginal effects evaluated at the mean of the independent variables
of the sample into households not exposed to
are reported for probit estimates. Huber-White robust standard errors (clustered at the circle level) are presented, significantly the survey in the first round. However, the different than zero at * 90%, ** 95%, *** 99% confidence.
Economic Political Weekly
EPW
SPECIAL ARTICLE
regressions of the impact of receiving the IEC and household survey within administrative circles, using circle fixed- effects. The results confirm that the IEC had no impact on enrolment, households sampled for the household survey were 8-9 percentage points more likely to enrol, and households in both the IEC and household survey groups were an additional 5 percentage points more likely to enrol relative to control households that received neither the IEC nor the household survey.
All effects were on the extensive enrolment margin; conditional on the household being enrolled, there was no further impact on the number of members enrolled (Table 1, column 3). Recall that under the RSBY policies, insurance companies have full incentives to enrol the first member of the household (the head) and zero incentives to enrol any additional members. Given that households who were part of the IEC and the household survey received explicit information about the number of members who could be enrolled and were informed that household members could not be enrolled unless they were physically present at the enrolment site, we thought it likely that the intervention would have resulted in greater participation of household members in enrolled households. Ultimately, we find no such impact.
That the IEC had little effect on enrolment may be related to the implementation context discussed in Section 2. Due to delays in registration, the IEC took place several months before RSBY cards were available, and because of the uncertainty of the rollout timeline, the IEC was unable to provide households with firm information on when enrolment would take place in their areas. As many rumours were circulating about the type of benefits provided by the RSBY card, the IEC, by providing accurate information, may have reduced households’ incentives to enrol to receive only one specific benefit, i e, hospitalisation insurance. As the household survey ended up being fielded in the midst of the enrolment period, the information provided through the survey questionnaire itself and through any discussion with the enumerators may have been more salient for household enrolment decisions. However, obtaining a “double dose” of information in which the message from the earlier IEC was reinforced through the household survey also seems to have been a significant influence on households’ enrolment decisions.
4.2 Utilisation
impact on claims rates for households that were sampled for both the IEC and the household survey.
The IEC may have ultimately induced households who were healthier to enrol in the RSBY, even if the campaign did not impact overall enrolment rates.9 Under such a scenario, households who enrolled after receiving the IEC would have been less likely to make claims as they would have fewer underlying health problems that would require hospitalisation. Unfortunately, the administrative data available on enrolled households contains little information that can be used to assess the pre-existing health s tatus of households. Using the administrative data, we find no significant differences between the characteristics of households sampled and not sampled for the IEC in terms of the number of household members who enrolled under each policy, the age of enrolled household members, the number of females, the number of elderly, and the number of children and young children ( results not shown).10
4.3 Profits
Given the difference in claims rates induced by the IEC, we examine whether it could be profitable for insurance companies to run information and education campaigns themselves. Premiums paid by the government to the insurance company under the RSBY programme are constant and thus do not vary by the likelihood that a claim will be incurred. Insurance companies cannot turn away sicker households, but have strong incentives to enrol as many (and as healthy) households as possible to reduce their average claims risk. An IEC might be a successful way to induce households with lower expected claims rates to enrol.11
Insurance company profits are determined by total revenue (premium*number of enrolled households) minus total claims and administrative costs. We calculate a rough measure of insurance company profits for each group of households, setting administrative costs to zero for all groups for convenience. Figure 2 reports these net profit measures, normalised by dividing by the number of policies in each group. Excluding the administrative costs of running the campaign, an IEC appears to be profitable for insurance companies – the net profit per enrolled household increases substantially for households that received the IEC and/or the household survey.12 This result is confirmed in
regression analysis that Figure 2: Net Profits across Groups of Households controls for variation
400
across administrative
| While the IEC did not have a significant impact on enrolment | circles – net profits |
| rates, the campaign did have an impact on utilisation rates | are Rs 74 higher for |
| among households that ultimately enrolled. About 4.6% of en | each policy issued to |
| rolled households that received the IEC (and 4% of households | a household sampled |
| that received both the IEC and the household survey) have at | for the IEC (Table 1, |
| least one transaction under the RSBY during the policy period. As | column 7). In the re |
| shown in Figure 1, this utilisation rate is 22% to 30% lower than | gression analysis, net |
| households that did not receive the IEC (of whom 6% had at least | profits are not signi |
| one transaction). These differences remain significant after ac | ficantly higher for |
| counting for circle differences; as shown in Table 1 (columns 4 | the household survey |
| and 5), households sampled for the IEC have significantly lower | sample even assuming |
| claim rates at 1.30 to 1.46 percentage points below comparison | a zero cost of providing |
| group households. The household survey, however, did not have | information through |
| a significant impact on claims rates, nor is there a differential | this channel. |
Net profits per policy (Rs)
300
200
100
0
Control IEC HH IEC+HH Survey Survey
The figure shows insurance company profits (the premium minus average claimed amounts) from administrative data for four groups of households: the control group that received neither an IEC nor a household survey; the group that received only the IEC, the group that received only the household survey and the group that received both. Profits were higher for the groups that received the IEC, consistent with the lower claims reported for this group in Figure 1.
august 6, 2011 vol xlvi no 32
EPW4.4 Characteristics of Enrolled Households
The data collected from the household survey can be used to look at the characteristics of households who chose to enrol in the RSBY and whether the IEC had an impact on the type of households that enrolled in this sample. However, as the administrative data shows that administering the household survey had a direct impact on enrolment outcomes; these results require the significant caveat that household decisions in the survey sample may be different from the majority of households that did not r eceive the survey.
The household survey suggests that the IEC had a significant impact on increasing pre-existing knowledge about the RSBY scheme at the time of the household survey – only around 32% of households reported knowing about the RSBY, but households in the IEC sample were 5-6 percentage points more likely to have heard about the RSBY. Not surprisingly, knowledge about the RSBY is an important determinant of enrolment – households that claimed to know about RSBY were 54 percentage points, or more than twice as likely, to enrol (Table 2). Examining other household characteristics, the targeting of the programme appears to be successful. Poorer households are significantly more likely to enrol in the RSBY, as are households with more children – each additional child increases the probability of enrolment by 1-2 percentage points.
Table 2 shows that household experience with health shocks and other risk-coping mechanisms also seem to be important
Table 2: Correlations between Household Health Status and Enrolment
| Independent Variable: | Marginal Effect on | Mean of dep |
|---|---|---|
| Enrolment Rate | var in Sample | |
| (Std Err) | (Std Dev) | |
| Respondent has heard about RSBY (1=yes) | 0.542 *** | 0.32 |
| (0.027) | ||
| Household has prior experience with health insurance | 0.577 *** | 0.26 |
| (0.038) | ||
| Household earnings in the last week (Rs) | -0.00006 *** | 886.98 |
| (0.000) | (693.20) | |
| Highest education level in household | -0.0053 | 8.98 |
| (0.005) | (3.06) | |
| Number of household members | -0.0026 | 5.50 |
| (0.003) | (2.30) | |
| Number of children | 0.015 ** | 2.21 |
| (0.007) | (1.65) | |
| Number of over 60s | -0.040 | 0.41 |
| (0.025) | (0.64) | |
| Spent at least Rs 500 on health last year | 0.041 *** | 0.27 |
| (0.013) | ||
| Borrowed to pay for health expenses last year | 0.051 ** | 0.21 |
| (0.021) | ||
| Death of a household member in the last year | -0.133 *** | 0.20 |
| (0.017) | ||
| Household head has a self-reported health problem | -0.081 * | 0.09 |
| (0.042) | ||
| Household member has trouble lifting a 5 Kg bag | 0.0057 | 0.51 |
| (0.022) | ||
| Expect that a household member will require hospitalisation | 0.044 * | 0.18 |
| (0.026) | ||
| Household has high discount factor | 0.072 *** | 0.37 |
| (0.020) |
All regressions include administrative circle-fixed effects and an indicator for the PSI intervention. Marginal effects are presented for probit estimates. Huber-White robust standard errors (clustered at the circle level) are presented, significantly different than zero at * 90%, ** 95%, *** 99% confidence.
Economic Political Weekly
EPW
determinants of enrolment. Households that have prior experience with health insurance are vastly more likely to take up the RSBY, as are those with a high discount factor. Additionally, households that have spent at least Rs 500 on healthcare e xpenses in the last year or that report having borrowed money in the past year in order to pay for healthcare expenses are 4-5 percentage points more likely to enrol. To the extent that households have adaptive expectations and base their expectations of future hospitalisation on past experiences, those that experienced bad health outcomes should be more likely to enrol. Households that have experienced a large negative shock in the form of a recent death in the family are less likely to enrol, however. This result makes sense if households anticipate that their future needs for hospitalisation will be lower immediately following the death of a (potentially sick) household member.
The household survey sample does not provide evidence or “cream-skimming” households. Examining a wide range of indicators of current household health status shows that only about one in ten measures is a marginally significant predictor of enrolment; whether a household member has a severe or recurrent health problem and indicators of health status measured using activities of daily living are not significantly correlated with enrolment. There is some indication that households where the head self-reports a health problem are 8 percentage points less likely to enrol, but such a result is expected purely by chance. F inally, there is suggestive evidence that households predicting future hospitalisations are a little more likely to enrol, again contrary to our concern that insurance companies may be able to “select” healthier individuals.
While some household characteristics are correlated with the probability of enrolment, we find few significant differences in these characteristics between enrolled households that received and did not receive the IEC. For instance, households with lower earnings were not more likely to enrol if they received the IEC, nor were households with more members. At most, there is some suggestive evidence that households that had suffered from – and paid for – health shocks in the past may have made different e nrolment decisions when they received the IEC. In particular, households that reported borrowing to pay for health expenses in the last year were about 8 percentage points more likely to enrol if they received the IEC. Also, households where a household member had died in the last year were even less likely to enrol if they received the information campaign; perhaps these households have a lower anticipated need for hospitalisation.13 The IEC might thus have some success at convincing households to enrol using an approach modelled on adaptive expectations.
In combination, the results from the household survey sample suggest that households that enrol in the RSBY are on average poorer and have faced health shocks in the past and that the marginal households induced to enrol by the addition of the IEC are no less healthy than other enrolled households. Although past health shocks and current perceptions of health status are correlated with enrolment decisions, these factors do not have much predictive power on whether households will make a claim d uring the following policy year,14 which is important from the perspective of insurance company profit motives.
SPECIAL ARTICLE
5 Discussion
Our results have implications for both the initial experience of the RSBY implementation in the state of Delhi and for the potential design of evaluations of the larger programme. To begin with, a key issue in the administration of the RSBY is whether i nsurance companies will be able to “game” the system by insuring primarily relatively healthy households. For insurance companies, as well as for those seeking to maintain political support for the programme, there is also a question of how enrolments can be increased beyond those seen in the first year of implementation; in many such discussions, IECs have been promoted as a tool for increasing enrolment.
On the positive side, our results show little evidence of “cream-skimming” by the insurer with some evidence that those who had suffered health shocks in the past were more likely to enrol. Also on the positive side, the results suggest that even in states where the supply-side administration of the programme is poor, some kinds of information can dramatically increase e nrolment. The household survey itself increased enrolment by 9 percentage points, and households exposed both to the IEC and the household survey were 15 percentage points or 60% more likely to enrol. Further, the marginal household enrolled as a result of information provision was healthier with lower claims relative to the average household. Thus, information provision can also lower insurance company liabilities by adding “good risks” to the enrolment pool.
On the negative side, we know no more than when we started about what constitutes a good IEC. Enormous effort was put into designing the IEC campaign and careful cost estimates of the campaign were made for future use. However, this IEC campaign had no impact on its own. We do not know precisely what information was conveyed to households through the household s urvey (which did increase enrolment) and what form this information took; given our relative ignorance, we are unable to price an effective IEC.
A second key issue on which this paper has bearing is how to design an evaluation of the programme that will allow researchers to understand the impact of the RSBY on financial protection and health outcomes. The crucial result here is that conducting a household survey is likely to increase the salience of the programme among surveyed households; this in turn could increase participation in the programme. Accounting for this Hawthorne effect, we propose two alternate designs. We feel that these d esigns reduce the risk of conducting an evaluation in a context with possibly poor implementation while maintaining an emphasis on causal impacts, at least in the context of instrumental variables estimators.
In the first design, randomly selected households (or villages) are “encouraged” to participate in the programme, where the e ncouragement could consist of financial subsidies, information or other enabling/incentivising mechanisms. The targeted households/villages are then tracked through the administrative data and the impact of the encouragement intervention on enrolment is verified. If the encouragement is shown to be successful and power calculations show that the size of the effect is sufficiently large, an end-line household survey can be implemented at the conclusion of the policy period. Such a survey would include the treated households/villages as well as a set of households/ villages that were not exposed to the encouragement. These control villages could be part of a design where there is random allocation into treatment and control or where more sophisticated methods of matching are also used to increase efficiency. Results from the end-line survey of both treatment and controls can then be used to evaluate the impact of the programme using standard tools from the literature, in particular ITT and instrumental variables estimators. Note that in this design, no household survey is conducted until after the encouragement is verified to have had a sufficiently large effect on the outcomes of interest in administrative data. To the extent that household characteristics that are of interest in the analysis are expected to be static (e g, education of household head) or may be obtained with a reasonable recall (e g, death in the family in the last year), collecting household data at the end-line may not pose any significant difficulties for subsequent evaluations.
REVIEW OF WOMEN’S STUDIES
April 30, 2011
Women and Water: Issues of Gender, Caste, Class and Institutions – Maithreyi Krishnaraj Questioning Masculinities in Water – Margreet Zwarteveen ‘They Are Not of This House’: The Gendered Costs of Drinking Water’s Commodification – Kathleen O’Reilly Caste, Gender and the Rhetoric of Reform in India’s Drinking Water Sector – Deepa Joshi Women and Decentralised Water Governance: Issues, Challenges and the Way Forward – Seema Kulkarni
For copies write to: Circulation Manager,
Economic and Political Weekly,
320-321, A to Z Industrial Estate, Ganpatrao Kadam Marg, Lower Parel, Mumbai 400 013. email: circulation@epw.in
august 6, 2011 vol xlvi no 32
In the second design, the household survey itself is used as the outcomes) and baseline surveys of households are widely used to encouragement, perhaps coupled with further incentive mecha-understand the heterogeneous impact of a treatment. In this nisms. If the encouragement effect from the baseline survey is study, the administrative data verifies that we cannot observe verified to be sufficiently large using the administrative data, a baseline characteristics without altering programme particisecond household survey could then be conducted. As in the first pation. Household surveys are expensive and in order to detect design, the second, end-line household survey will also have to impacts on hospitalisations, which are relatively rare events, sample additional households not surveyed in the baseline sur-an evaluation of the impact of the programme on household vey – these additional households will serve as the control group health and financial protection outcomes would require a housethat did not receive the encouragement intervention. The advan-hold sample exceeding 25,000. We feel that there is substantial tage of this design is that there is at least some information on the risk involved in conducting baseline surveys for RSBY evaluabaseline characteristics of households; this design may be parti-tions, as in the second design, because significant resources will cularly relevant when the characteristics of interest are expected be committed in a context where there is considerable uncerto be altered by the programme and thus baseline information on tainty over whether an encouragement intervention will yield these characteristics is useful for subsequent analysis. a large enough effect for tractable analysis on household
In standard programme evaluation, the Hawthorne effect is outcomes. U nder the first design, the ex ante investment is thought to be less severe (although without independent sources restricted to the cost of the encouragement intervention, and of data on outcomes of interest, like that collected through the subsequent resources can be committed only after the first stage RSBY administrative data, very few studies have actually veri-effect is verified with no risk of introducing distortionary
fied the potential impact of Hawthorne effects on evaluation Hawthorne effects.
Notes
1 See Palacios (2010) for further description of the scheme and the early implementation experience.
2 This could also be attributed to a dated BPL list data. See Sun (2010) for a detailed analysis of enrolment patterns.
3 In the context of designing evaluations of the RSBY programme, these uncertainties appear to be part and parcel of the overall implementation. Thus any evaluation strategy should be robust to a less than perfect situation on the ground – an observation similar to that of evaluations of government programmes in the United States in the 1980s and 1990s (Moffitt 2004).
4 See Grover and Palacios (2010) for further analysis of the enrolment experience in Delhi.
5 Where state IEC activities were better conducted, they probably did make a difference; enrolment rates across circles varied dramatically ranging from a low of 4% to a high of 88%. Informal conversations suggested that these differences may have been correlated with the interest and capacity of the PDS officer. In some cases, these officers went out of the way to ensure that lists of eligible beneficiaries were posted at the enrolment office and that posters of the programme were prominently displayed.
6 See Grover and Palacios (2010).
7 Sun (2010).
8 In Delhi, the initial sample list contained prima
rily the poorest households, known as Antyodaya or AAY households. The initial list of beneficiaries was later supplemented with additional lists of BPL households. However, all sampling for the IEC and household survey was done using the initial list, and all results are thus reported using the initial sample frame rather than the ultimate list of eligible households.
9 Of course, the IEC may have impacted utilisation rates through channels other than changing the composition of enrolled households. For instance, the information provided through the campaign may have misled households in a way that reduced utilisation rates. To take a potentially plausible (and non-malicious) example, if Hospital X, which is locally the preferred treatment option, was empanelled after the IEC but before enrolment, and if during the campaign households asked if the RSBY card could be used at Hospital X, the answer would, at that time, have been no. If updated information on empanelled hospitals was not made widely available, households may not have been aware of the change and may not have used their
Economic Political Weekly august 6, 2011
EPWcard when seeking treatment at Hospital X.
10 The household survey data is used to examine the health status of enrolled and non-enrolled households in the following sub-section, but because the household survey itself induced a change in the enrolment rates, households that completed the survey may look very different from households that did not. Thus, caution needs to be taken in interpreting the likely non-generalisable results from the household survey sample.
11 Of course, an IEC administered by an insurance company would need to be monitored to ensure that the information provided was accurate and that the IEC was not used to manipulate the likelihood that households with higher claims risk enrolled.
12 Households that received the IEC have marginally higher value claims than other households (Table 1, Column 6), but this difference is overwhelmed by the differences in the claims rate.
13 On the other hand, households where at least one member reported having significant difficulty lifting objects were 4 percentage points more likely to enrol when they received the IEC, though of all the household health characteristics examined, this is the only one that had a significant impact.
14 Results available on request from authors.
References
Berman, Peter, Rajeev Ahuja and Laveesh Bhandari (2010): “The Impoverishing Effect of Healthcare Payments in India: New Methodology and Findings”, Economic Political Weekly, Vol XLV(16): 65-71.
Das, J and J Hammer (2007): “Money for Nothing: The Dire Straits of Medical Practice in Delhi, I ndia”, Journal of Development Economics, Vol 83(1): 1-36.
Desai, S (2009): “Keeping the ‘Health’ in Health Insurance”, Economic Political Weekly, 19 September, Vol XLIV(38).
Garg, C and A Karan (2009): “Reducing Out-of-Pocket Expenditures to Reduce Poverty: A Disaggregated Analysis at Rural-Urban and State Level in India”, Health Policy and Planning, Vol 24(2): 116-28.
Government of India (2005): “Report of the National Commission on Macroeconomics and Health”, available at http://www.who.int/macrohealth/ action/Report%20of%20the%20National%20 Commission.pdf
vol xlvi no 32
Grover, S and R Palacios (2010): “The First Two Years of RSBY in Delhi”, RSBY Working Paper Series, #2, October, Ministry of Labour and Employment, Government of India, New Delhi (www.rsby.gov.in).
King, G, E Gakidou, K Imai, J Lakin, R T Moore, C Nall, N Ravishankar, M Vargas, M Maria Tellez-Rojo, J E Hernandez Avila, M Hernandez Avila and H Hernandez Llamas (2009): “Public Policy for the Poor? A Randomised Assessment of the Mexican Universal Health Insurance Programme”, The Lancet, Vol 373(9673): 1447-54.
Mahal Ajay, Abdo S Yazbeck, David H Peters and G N V Ramana (2001): “The Poor and Health Service Use in India”, The World Bank, Washington DC.
Moffitt, R (2004): “The Role of Randomised Field T rials in Social Science Research: A Perspective from Evaluations of Reforms of Social Welfare Programs”, American Behavioral Scientist, Vol 47(5): 506-40.
Palacios, R (2010): “A New Approach to Providing Health Insurance to the Poor in India: The Early Experience of Rashtriya Swasthya Bima Yojana”, RSBY Working Paper Series, #1, September, Ministry of Labour and Employment, Government of India, New Delhi (www.rsby.gov.in).
Selvaraj, S and A Karan (2009): “Deepening Health Insecurity in India: Evidence from National Sample Surveys since 1980s”, Economic Political Weekly, 3 October, Vol XLIV(40).
Sun, C (2010): “An Analysis of RSBY Enrolment Patterns: Preliminary Evidence and Lessons from the Early Experience”, RSBY Working Paper Series, #2, September, Ministry of Labour and Employment, Government of India, New Delhi (www.rsby.gov.in).
Thornton, Rebecca, Laurel E Hatt, Erica Field, Mursaleena Islam, Freddy Solís, Martha Azuzena and González Moncada (2010): “Social Security Health Insurance for the Informal Sector in Nicaragua: A Randomised Evaluation”, Health Economics, Vol 19(S1): 181-206.
World Health Organisation (2000): The World Health Report: Health Systems: Improving Performance (Geneva: WHO).
– (2008): World Health Statistics (Geneva: WHO). Zwane, Alix Peterson, Jonathan Zinman, Eric Van Dusen, William Pariente, Clair Null, Edward
Miguel, Michael Kremer, Dean S Karlan, Richard Hornbeck, Xavier Giné, Esther Duflo, Florencia Devoto, Bruno Crepon and Abhijit Banerjee (2011):
“Being Surveyed Can Change Later Behaviour and Related Parameter Estimates”, PNAS, 108(5) 1821-26.
