-
From theory to 'measurement' in complex
interventions: Methodological lessons from the
development of an e-health normalisation
instrument
Background:
Although empirical and theoretical understanding of processes of implementation in healthcare is advancing, translation of theory into structured measures that capture the complexinterplay between interventions, individuals and context remain limited. This paper aimed to(1) describe the process and outcome of a project to develop a theory-based instrument formeasuring implementation processes relating to e-health interventions; and (2) identify keyissues and methodological challenges for advancing work in this field.
Methods:
A 30-item instrument (Technology Adoption Readiness Scale (TARS)) for measuringnormalisation processes in the context of e-health service interventions was developed on thebasis on Normalization Process Theory (NPT). NPT focuses on how new practices becomeroutinely embedded within social contexts. The instrument was pre-tested in two health caresettings in which e-health (electronic facilitation of healthcare decision-making and practice)was used by health care professionals.
Results:
The developed instrument was pre-tested in two professional samples (N = 46; N = 231).Ratings of items representing normalisation 'processes' were significantly related to staffmembers' perceptions of whether or not e-health had become 'routine'. Key methodologicalchallenges are discussed in relation to: translating multi-component theoretical constructs intosimple questions; developing and choosing appropriate outcome measures; conductingmultiple-stakeholder assessments; instrument and question framing; and more general issuesfor instrument development in practice contexts.
Conclusions:
To develop theory-derived measures of implementation process for progressing research inthis field, four key recommendations are made relating to (1) greater attention to underlyingtheoretical assumptions and extent of translation work required; (2) the need for appropriatebut flexible approaches to outcomes measurement; (3) representation of multiple perspectivesand collaborative nature of work; and (4) emphasis on generic measurement approaches thatcan be flexibly tailored to particular contexts of study.
-
A review of statistical estimators for risk-adjusted
length of stay: analysis of the Australian and new
Zealand intensive care adult patient data-base,
2008-2009
Background:
For the analysis of length-of-stay (LOS) data, which is characteristically right-skewed, anumber of statistical estimators have been proposed as alternatives to the traditional ordinaryleast squares (OLS) regression with log dependent variable.
Methods:
Using a cohort of patients identified in the Australian and New Zealand Intensive CareSociety Adult Patient Database, 2008-2009, 12 different methods were used for estimation ofintensive care (ICU) length of stay. These encompassed risk-adjusted regression analysis offirstly: log LOS using OLS, linear mixed model [LMM], treatment effects, skew-normal andskew-t models; and secondly: unmodified (raw) LOS via OLS, generalised linear models[GLMs] with log-link and 4 different distributions [Poisson, gamma, negative binomial andinverse-Gaussian], extended estimating equations [EEE] and a finite mixture model includinga gamma distribution. A fixed covariate list and ICU-site clustering with robust variance wereutilised for model fitting with split-sample determination (80%) and validation (20%) datasets, and model simulation was undertaken to establish over-fitting (Copas test). Indices ofmodel specification using Bayesian information criterion [BIC: lower values preferred] andresidual analysis as well as predictive performance (R2, concordance correlation coefficient(CCC), mean absolute error [MAE]) were established for each estimator.
Results:
The data-set consisted of 111663 patients from 131 ICUs; with mean(SD) age 60.6(18.8)years, 43.0% were female, 40.7% were mechanically ventilated and ICU mortality was 7.8%.ICU length-of-stay was 3.4(5.1) (median 1.8, range (0.17-60)) days and demonstrated markedkurtosis and right skew (29.4 and 4.4 respectively). BIC showed considerable spread, from amaximum of 509801 (OLS-raw scale) to a minimum of 210286 (LMM). R2 ranged from 0.22(LMM) to 0.17 and the CCC from 0.334 (LMM) to 0.149, with MAE 2.2-2.4. Superiorresidual behaviour was established for the log-scale estimators. There was a general tendencyfor over-prediction (negative residuals) and for over-fitting, the exception being the GLMnegative binomial estimator. The mean-variance function was best approximated by aquadratic function, consistent with log-scale estimation; the link function was estimated(EEE) as 0.152(0.019, 0.285), consistent with a fractional-root function.
Conclusions:
For ICU length of stay, log-scale estimation, in particular the LMM, appeared to be the mostconsistently performing estimator(s). Neither the GLM variants nor the skew-regressionestimators dominated.
-
Intraclass reliability for assessing how well Taiwan
constrained hospital-provided medical services using
statistical process control chart techniques
Background:
Few studies discuss the indicators used to assess the effect on cost containment in healthcareacross hospitals in a single-payer national healthcare system with constrained medicalresources. We present the intraclass correlation coefficient (ICC) to assess how well Taiwanconstrained hospital-provided medical services in such a system.
Methods:
A custom Excel-VBA routine to record the distances of standard deviations (SDs) from thecentral line (the mean over the previous 12 months) of a control chart was used to constructand scale annual medical expenditures sequentially from 2000 to 2009 for 421 hospitals inTaiwan to generate the ICC. The ICC was then used to evaluate Taiwan's year-basedconvergent power to remain unchanged in hospital-provided constrained medical services. Abubble chart of SDs for a specific month was generated to present the effects of using controlcharts in a national healthcare system.
Results:
ICCs were generated for Taiwan's year-based convergent power to constrain its medicalservices from 2000 to 2009. All hospital groups showed a gradually well-controlled supply ofservices that decreased from 0.772 to 0.415. The bubble chart identified outlier hospitals thatrequired investigation of possible excessive reimbursements in a specific time period.
Conclusion:
We recommend using the ICC to annually assess a nation's year-based convergent power toconstrain medical services across hospitals. Using sequential control charts to regularlymonitor hospital reimbursements is required to achieve financial control in a single-payernationwide healthcare system.
-
Bridging knowledge translation gap in health in
developing countries: visibility, impact and
publishing standards in journals from the Eastern
Mediterranean
Background:
Local and regional scientific journals are important factors in bridging gaps in healthknowledge translation in low-and middle-income countries. We assessed indexing, citationsand publishing standards of journals from the Eastern Mediterranean region.
Methods:
For journals from 22 countries in the collection of the Index Medicus for the EasternMediterranean Region (IMEMR), we analyzed indexing in bibliographical databases andcitations during 2006-2009 to published items in 2006 in Web of Science (WoS) andSCOPUS. Adherence to editorial and publishing standards was assessed using a specialchecklist.
Results:
Out of 419 journals in IMEMR, 19 were indexed in MEDLINE, 23 in WoS and 46 inSCOPUS. Their impact factors ranged from 0.016 to 1.417. For a subset of 175 journals withavailable tables of contents from 2006, articles published in 2006 from 93 journals received2068 citations in SCOPUS (23.5% self-citations) and articles in 86 journals received 1579citations to in WoS (24.3% self-citations) during 2006-2009. Citations to articles camemostly from outside of the Eastern Mediterranean region (76.8% in WoS and 75.4% inSCOPUS). Articles receiving highest number of citations presented topics specific for theregion. Many journals did not follow editorial and publishing standards, such addressingrequirements about the patient's privacy rights (68.0% out of 244 analyzed), policy onmanaging conflicts of interest (66.4%), and ethical conduct in clinical and animal research(66.4%).
Conclusion:
Journals from the Eastern Mediterranean are visible in and have impact on global scientificcommunity. Coordinated effort of all stakeholders in journal publishing, includingresearchers, journal editors and owners, policy makers and citation databases, is needed tofurther promote local journals as windows to the research in the developing world and thedoors for valuable regional research to the global scientific community.
-
Quality of DNA extracted from saliva samples collected with the OrageneTM kit DNA self-collection kit
Background:
Large epidemiological studies in DNA biobanks have increasingly used less invasive methods for obtaining DNA samples, such as saliva collection. Although lower amounts of DNA are obtained as compared with blood collection, this method has been widely used because of its more simple logistics and increased response rate. The present study aimed to verify whether a storage time of 8 months decreases the quality of DNA from collected samples.
Methods:
Saliva samples were collected with an OrageneTM DNA Self-Collection Kit from 4,110 subjects aged 14-15 years. The samples were processed in two aliquots with an 8-month interval between them. Quantitative and qualitative evaluations were carried out in 20% of the samples by spectrophotometry and genotyping. Descriptive analyses and paired t-tests were performed.
Results:
The mean volume of saliva collected was 2.2mL per subject, yielding on average 184.8ug DNA per kit. Most samples showed an absorbance ratio (RAT) between 1.6 and 1.8 in the qualitative evaluation. The evaluation of DNA quality by TaqMan(R), High Resolution Melting (HRM), and restriction fragment length polymorphism-PCR (RFLP-PCR) showed a rate of success of up to 98% of the samples. The sample storage time did not reduce either the quantity or quality of DNA extracted with the Oragene kit.
Conclusion:
The study results showed that a storage period of 8 months at room temperature storage did not reduce the quality of the DNA obtained. In addition, the use of the Oragene kit during fieldwork in large population-based studies allows for DNA of high quantity and high quality.
-
Systematic review of methods used in meta-analyses where a primary outcome is an adverse or unintended event
Background:
Adverse consequences of medical interventions are a source of concern, but clinical trials may lack power to detect elevated rates of such events, whereas observational studies have inherent limitations. Meta-analysis allows the combination of individual studies, which can increase power and provide stronger evidence relating to adverse events. However, meta-analysis of adverse events has associated methodological challenges. The aim of this study was to systematically identify and review the methodology used in meta-analyses where a primary outcome is an adverse or unintended event, following a therapeutic intervention.
Methods:
Using a collection of meta-analyses identified previously, 166 individual meta-analysis references were selected for review. At least one of the primary outcomes in each meta-analysis study was an adverse or unintended event. The nature of the intervention, source of funding, number of individual meta-analyses performed, number of primary studies included in the review, and use of meta-analytic methods were all recorded. Specific areas of interest relating to the methods used included the choice of outcome metric, methods of dealing with sparse events, heterogeneity, publication bias and use of individual patient data.
Results:
The 166 meta-analysis references were published between 1994 and 2006. Interventions included drugs and surgery among other interventions. Many of the references being reviewed included multiple meta-analyses with 44.6\% (74/166) including more than ten. Randomised trials only were included in 42.2\% of meta-analyses (70/166), observational studies only in 33.7\% (56/166) and a mix of observational studies and trials in 15.7\% (26/166). Sparse data, in the form of zero events in one or both arms where the outcome was a count of events, was found in 64 meta-analysis references, of which 41 (64.1\%) had zero events in both arms.
Conclusions:
Meta-analyses of adverse events data are common and useful in terms of increasing the power to detect an association with an intervention, especially when the events are infrequent. However, with regard to existing meta-analyses, a wide variety of different methods have been employed, often with no evident rationale for using a particular approach. More specifically, the approach to dealing with zero events varies, and guidelines on this issue would be desirable.
-
Selecting a comparison group for 5-year oral and
pharyngeal cancer survivors: Two methods
Background:
To assess potential long-term consequences of cancer treatment, studies that includecomparison groups are needed. These comparison groups should be selected in a way thatallows the subtle long-range effects of cancer therapy to be detected and distinguishes themfrom the effects of aging and other risk factors. The purpose of this investigation was to testtwo methods of recruiting a comparison group for 5-year oral and pharyngeal cancersurvivors (peer-nominated and listed sample) with emphasis on feasibility and the quality ofthe match.
Methods:
Participants were drawn from a pool of 5-year survivors treated at a large Southeasternhospital. A peer-nominated sample was solicited from the survivors. A listed sample matchedon sex, age, and zip code was purchased. Telephone interviews were conducted by aprofessional call center.
Results:
The following represent our key findings: The quality of matching between survivors andlisted sample was better than that between survivors and peer-nominated group in age andsex. The quality of matching between the two methods on other key variables did not differexcept for education, with the peer method providing a better match for the survivors than thelisted sample. The yield for the listed sample method was greater than for the peer-nominatedMethodThe cost per completed interview was greater for the peer-nominated method thanthe listed sample.
Conclusion:
This study not only documents the methodological challenges in selecting a comparisongroup for studies examining the late effects of cancer treatment among older individuals butalso documents challenges in matching groups that potentially have disproportionate levels ofcomorbidities and at-risk health behaviors.
-
Including mixed methods research in systematic
reviews: Examples from qualitative syntheses in TB
and malaria control
Background:
Health policy makers now have access to a greater number and variety of systematic reviewsto inform different stages in the policy making process, including reviews of qualitativeresearch. The inclusion of mixed methods studies in systematic reviews is increasing, butthese studies pose particular challenges to methods of review. This article examines thequality of the reporting of mixed methods and qualitative-only studies.
Methods:
We used two completed systematic reviews to generate a sample of qualitative studies andmixed method studies in order to make an assessment of how the quality of reporting andrigor of qualitative-only studies compares with that of mixed-methods studies.
Results:
Overall, the reporting of qualitative studies in our sample was consistently better whencompared with the reporting of mixed methods studies. We found that mixed methods studiesare less likely to provide a description of the research conduct or qualitative data analysisprocedures and less likely to be judged credible or provide rich data and thick descriptioncompared with standalone qualitative studies. Our time-related analysis shows that for bothtypes of study, papers published since 2003 are more likely to report on the study context,describe analysis procedures, and be judged credible and provide rich data. However, thereporting of other aspects of research conduct (i.e. descriptions of the research question, thesampling strategy, and data collection methods) in mixed methods studies does not appear tohave improved over time.
Conclusions:
Mixed methods research makes an important contribution to health research in general, andcould make a more substantial contribution to systematic reviews. Through our carefulanalysis of the quality of reporting of mixed methods and qualitative-only research, we haveidentified areas that deserve more attention in the conduct and reporting of mixed methodsresearch.
-
Assessing measurement invariance of familism and parental respect across race/ethnicity in adolescents
Background:
Familism and parental respect are culturally derived constructs rooted in Hispanic and Asian cultures, respectively. Measures of these constructs have been utilized in research and found to predict delays in substance use initiation and reduced levels of use. However, given that these measures are explicitly designed to tap constructs that are considered important by different racial/ethnic groups, there is a risk that the measurement properties may not be equivalent across groups.
Methods:
This study evaluated the measurement equivalence of measures of familism and parental respect in a large and diverse sample of middle school students in Southern California (n = 5646) using a multiple group confirmatory factor analysis approach.
Results:
Results showed little evidence of measurement variance across four racial/ethnic groups (African American, Hispanic, Asian, and non-Hispanic White), supporting the continued use of these measures in diverse populations. Some differences between latent variable means were identified - specifically that the Hispanic group and the white group differed on familism.
Conclusions:
No evidence of invariance was found. However, the item distributions were highly positively skewed, indicating a tendency for youth to endorse the most positive response, which may reduce the reliability of the measures and suggests that refinement is possible.
-
Accuracy and precision of variance components in
occupational posture recordings: a simulation study
of different data collection strategies
Background:
Information on exposure variability, expressed as exposure variance components, is of vitaluse in occupational epidemiology, including informed risk control and efficient study design.While accurate and precise estimates of the variance components are desirable in such cases,very little research has been devoted to understanding the performance of data samplingstrategies designed specifically to determine the size and structure of exposure variability.The aim of this study was to investigate the accuracy and precision of estimators of betweensubjects,between-days and within-day variance components obtained by sampling strategiesdiffering with respect to number of subjects, total sampling time per subject, number of daysper subject and the size of individual sampling periods.
Methods:
Minute-by-minute values of average elevation, percentage time above 90degrees and percentagetime below 15degrees were calculated in a data set consisting of measurements of right upper armelevation during four full shifts from each of 23 car mechanics. Based on this parent data,bootstrapping was used to simulate sampling with 80 different combinations of the number ofsubjects (10, 20), total sampling time per subject (60, 120, 240, 480 minutes), number of daysper subject (2, 4), and size of sampling periods (blocks) within days (1, 15, 60, 240 minutes).Accuracy (absence of bias) and precision (prediction intervals) of the variance componentestimators were assessed for each simulated sampling strategy.
Results:
Sampling in small blocks within days resulted in essentially unbiased variance components.For a specific total sampling time per subject, and in particular if this time was small,increasing the block size resulted in an increasing bias, primarily of the between-days and thewithin-days variance components. Prediction intervals were in general wide, and even moreso at larger block sizes. Distributing sampling time across more days gave in general moreprecise variance component estimates, but also reduced accuracy in some cases.
Conclusions:
Variance components estimated from small samples of exposure data within working daysmay be both inaccurate and imprecise, in particular if sampling is laid out in largeconsecutive time blocks. In order to estimate variance components with a satisfying accuracyand precision, for instance for arriving at trustworthy power calculations in a plannedintervention study, larger samples of data will be required than for estimating an exposuremean value with a corresponding certainty.
|
