Quality Assessment - Science topic

Self-made and self-tailored risk of bias tool are common to be used, especially for studies that might don't have any suitable risk of bias, such as in-vitro studies. However, it might be better to properly look for any available RoB tool or most closely related

SERVQUAL model is an effective model for assessing the outcome of behavioural intention through mediating effect of satisfaction. Kindly visits the links.

Regards,

You can mark 1 point for every yes and 0 for no; (although there are other options like CD-cannot determine, NR-not reported ) and score the same out of 14 points in Microsoft excel. The grading will be decided on the total score: 0-5 (Poor) , 6-10 (fair) and 11-14 (Good)

Hope it helps

Interesting...following

Thank you

You should try The Spiritual Distress Assessment Tool. I hope it helps in your search. Good luck

You can't use one version of the Newcastle Ottawa Scale for all types of studies. I recommend using the specific version for each type.

Look at the journal guidelines ( the journal you are interested in submitting) to be sure- e.g. some require PRISMA for systematic reviews

Yes, there is quite a number of them.

Kindly, find the below article that might help you.

SANRA—a scale for the quality assessment of narrative review articles

Published online 2019 Mar 26. doi: 10.1186/s41073-019-0064-8

1. Go to Google Scholar

2. google health-topics/study-quality-assessment-tools

it will list on the top of the lists.

3.then select citation sign : ,,"

it will show you many referencing styles

For MLA

National Heart, Lung, and Blood Institute. "Study Quality Assessment Tools [https://www. nhlbi. nih. gov/health-topics/study-quality-assessment-tools]." (2019).

For APA

National Heart, Lung, and Blood Institute. (2019). Study Quality Assessment Tools [https://www. nhlbi. nih. gov/health-topics/study-quality-assessment-tools].

For Chicago

National Heart, Lung, and Blood Institute. "Study Quality Assessment Tools [https://www. nhlbi. nih. gov/health-topics/study-quality-assessment-tools]." (2019).

For Harvard

National Heart, Lung, and Blood Institute, 2019. Study Quality Assessment Tools [https://www. nhlbi. nih. gov/health-topics/study-quality-assessment-tools].

For Vancouver

National Heart, Lung, and Blood Institute. Study Quality Assessment Tools [https://www. nhlbi. nih. gov/health-topics/study-quality-assessment-tools].

Thank you all. I made up my mind and we are using the NIH quality assessment tools.

Thanks

other scoring systems are:

the Agency for Healthcare Research and Quality (ARHQ) methodology checklist

Quality Assessment of Diagnostic Accuracy Studies‐2 (QUADAS‐2) tool

Oxford Quality Scoring System

For interventional studies, you may use ROBIN-I tool https://methods.cochrane.org/methods-cochrane/robins-i-tool

For observational studies, the Newcastle-Ottawa Scale (NOS) in which there are slightly different details for case-control, cohort and cross-sectional studies can be used http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

Personally ROBIN-I is more comprehensive but not suitable for studies with no intervention.

Basically in quality assurance, there is still a debate as to "what quality should be and its appending quantitative and qualitative indicators". It is a big mistake to say "good, fair, poor" as quality is aimed at improvements and not as jugmental or prescriptive tool. What is good might not be good depending on the assessment metrics or in the "eyes of the beholders" or even in the capabilities of both the assessor and assessed. But using the performance excellence assessment framework of MBNQA for process criteria ADLI (approach, deployment, learning & integration) and results LeTCI (Levels, Trend, Comparisons & Integration) one can more objectively provide an objective score of where one stands rather than good/fair/poor standing.

can be isolated using different spectral signature between cloud cover and ice cover

CHAPTER 3. AIR QUALITY ASSESSMENT ... - EPA

https://www3.epa.gov/ttnecas1/regdata/RIAs/finalpbriach3.pdf

here's a good article on the issues

Also here's a review of a number of tools

https://www.rki.de/EN/Content/Institute/DepartmentsUnits/InfDiseaseEpidem/Div33/PRECEPT/Framework_Appendix_C.pdf?__blob=publicationFile

Ikhlas Almukahel ,

Thank you for your detailed answer. May I assume that you agree but there need to be more things for data quality assessment?

Agree with @ Roman Aleksander Tabisz. regards

Hello Godfrey,

Thank you for your suggestions. I appreciate it. I will take note of this.

Regards,

John

Int Clin Psychopharmacol. 2018 Jul;33(4):181-196.

Papola D1,2, Ostuzzi G1, Thabane L2, Guyatt G2, Barbui C1.

Hi Godfrey and Mathieu,

I do tend to agree, perhaps using different tools would be ideal.

I appreciate your advice and reference of tools.

Thanks again,

Jordan

Check out this article:

Real-Time Signal Quality-Aware ECG Telemetry System for IoT-Based Health Care Monitoring

It is possible, I've used it in my article.

I suggest to read its use through its official website.

Best regards,

Priastana

Hi Sanjay,

I'm wondering what characteristic  of linalool you wish to evaluate to measure the quality of.

Linalool can take different forms (S- or R-), resulting in different scent characteristics, or perhaps you are thinking of using linalool as something like a 'half-life'  marker of made tea age, measuring it's oxygen exposure >> degradation?

Lou Berkley

San Francisco

It seems useful:

Than you so much Craig and Tarang, 

Likert scale is 1 to 5 as you have there but one of the five i.e. the middle value is neutral. If you switch out moderate for neutral you have a likert scale. i.e the respondents can select very high, high, neutral (neither high or low) low or very low. 

Once you have all of your questionnaire results you can carry out a statistical analysis, if you have a good number of responses to work out. internal consistency and reliability of the results and how they relate to one another. 

Quantitative is basically trying to determine value. Quality checks inline within manufacturing you could argue are non value add. Its something you do to ensure you are doing something correctly. In an ideal world you should be doing things correctly and not require a check. 

That said Tacihi Ohno I believe talked selling quality. People are willing to pay more for a quality product. But its a difficult thing to put into terms of money. Unless you say by improving quality we can charge more (value) or by saying by improving the quality of a particular process we can eradicate another or a check by one man over 8 hour day. Therefore saving on the wage. Improving a manual job by implementing an automated job which gives a better quality output and therefore you can determine a pay back period for the machine by removing the workers. When the payback is complete you can determine a saving in terms of wages and potentially time and transfer this into money minus the machine overheads i.e. running cost and maintenance for instance.

Hope this helps

Martin  

Dear Sir. Concerning your issue about an external quality assessment program which is suitable for the entire adrenal steroid line. The progress in assay methodology, from the use of radioactive tracers to chemiluminescent signals, from competitive to chromatographic techniques and from serum or urine to saliva has considerably impacted on hormonal measurements. The clinician now may choose among multiple tests but the inherent variability in cortisol and ACTH secretion, coupled to lack of harmonization among assay procedures and normal ranges mandates careful interpretation of any result. LC-MS/MS now affords the specificity, imprecision, and limits of quantification necessary for the reliable measurement of steroids in human fluids, enhancing diagnostic capabilities, particularly when steroid profiles are available.  LC-MS/MS now affords the specificity, imprecision, and limits of quantification necessary for the reliable measurement of steroids in human fluids, thereby enhancing diagnostic capabilities, particularly when steroid profiles are available. Major advantages of tandem mass spectrometry include small sample size, the simultaneous measurement of many analytes, and enhanced specificity compared to IA methods. Mass spectrometric methods are still fairly labor intensive, and certainly require a higher level of laboratory expertise than do IA platforms. Occasional interferences when using mass spectrometric methods have been described, such as prednisolone/prednisone metabolite interference in urinary free cortisol measurements. It should be noted that currently reimbursement for steroid profile testing is not yet approved by Medicare (with the exception of the CAH steroid profile), nor are steroid profiles ordered frequently by clinicians. This could well change as steroid hormone profiling becomes more appreciated in the years ahead. Although mass spectrometric assays are not always more precise than IAs, they are more specific for measuring the analyte of interest. By omitting extraction and derivatization steps, the steroid tandem mass spectrometric procedures described here have good intrarun and interrun imprecision. Drug interference has been tested and found not to be a problem for the steroid and estrogen profiles reported in this review . These two methods are also relatively free of ion suppression. I think the following below links may help you in your analysis:

Thanks

SAR data quality - specifically SAR image quality - is pretty well described by

1) resolution

2) Impulse Response (Point Spread Function)

3) Additive Noise level

4) Multiplicative Noise Ratio

5) Spurious Signal Levels

6) Geometric Distortions

Some of these are a function of raw data quality, and some are a function of the image formation process...

Have a look on the Review Article published in Journal of Evidence-Based Medicine, the methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review (link).

The review authors reports:

1. The Cochrane Collaboration's tool for assessing risk of bias is the best available tool for assessing RCTs.

2. For cohort and case-control studies, the Newcastle-Ottawa Scale.

3. The Methodological Index for Non-Randomized Studies (MINORS) is an excellent tool for assessing non-randomized interventional studies.

4. The Agency for Healthcare Research and Quality (ARHQ) methodology checklist is applicable for cross-sectional studies.

5. For diagnostic accuracy test studies, the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool is recommended.

6. Assessment of Multiple Systematic Reviews (AMSTAR) is a measurement tool for systematic reviews/meta-analyses.

7. The Appraisal of Guidelines, Research and Evaluation (AGREE)-II instrument is widely used to evaluate clinical practice guidelines.

STROBE guideline is useful for assessing the reporting of observational studies [cross-sectional studies included]. It helps you answer the question, "Is the study properly reported?" It is INAPPROPRIATE for assessing the methodological quality of observational studies. A study may be of high quality methodologically BUT poorly reported such that it is difficult to replicate. Similarly, a study with poor methodological quality may be well-reported. I think the difference is clear (see: http://bmjopen.bmj.com/content/1/1/e000048.full)

Now, to assess methodological quality of cross-sectional studies, I think Newcastle-Ottawa. The NHLBI tool seems good too (seems like it borrows heavily from Newcastle-Ottawa) but I have not seen it used studies--could somebody share some? Thank you.

Hello Lloyd ,,, I am a member of the evidence based center , Cairo Univeristy Egypt and a memeber of CASP Egypt team ,,,,,,,I recommend using CASP checklist for quality assessment of cohort ans cross sectional studies or SIGN for the relevant study designs

link for CASP is http://www.casp-uk.net/

link for SIGN ishttp://www.sign.ac.uk/methodology/checklists.html

wish u all the best 

Hi Rania,

As mentioned in chapter 13 from the Cochrane Handbook, study designs are nowadays so diverse that it becomes difficult to label them with a single appellation. The best would be instead to evaluate specific design features (like: patient recruitment, treatment allocation, outcome assessment, etc). 

Regarding your questions:

An observational study is indeed observational: no treatment or intervention is provided by the researchers. Instead, features from the participants (such as risk factors for something) are collected and researchers try to establish an association between these factors and a specific outcome. They can start with the risk factors and follow participants over time to see if they develop the outcome (prospective design) or start with the outcome and look in the past if participants had risk factors (retrospective design).

An interventional study is a study where researchers assign participants to an intervention  (treatment, device, strategy, etc). You can have only one group (as it is the case in your beta-carotene study), or 2 (the most common design) or more. Treatment assignment can be randomised (determined by random chance) or not adequatelly randomised (quasi randomised) or determined by other factors. You can start with outcome assessment and determine if participants received the intervention in the past (retrospective design) or start with the intervention and see if participants develop the outcome (prospective design). 

Each specific design feature can be assessed with these tools. Having only 1 study group makes this feature at high risk of bias, but other features in this study might be at lower risk. You can still evaulate each of them using the appropriate tool (MOOSE for observational studies, Chapter 13 for interventional studies). 

Hope it helps

Frédérique

Below are tools that I have previously used, are easy to use, and some of them are recommended by Cochrane:

For qualitative studies: the CASP tool http://media.wix.com/ugd/dded87_29c5b002d99342f788c6ac670e49f274.pdf

For systematic reviews: the CASP tool http://media.wix.com/ugd/dded87_a02ff2e3445f4952992d5a96ca562576.pdf

For cross-sectional/survey studies: the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies: http://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort

For intervention studies: the EPHPP tool http://www.ephpp.ca/tools.html (first two files)

For risk of bias assessment for interventions: the EPOC criteria https://www.biomedcentral.com/content/supplementary/2046-4053-3-103-S2.pdf) 

For level of evidence assessment per outcome: the GRADE tool http://cccrg.cochrane.org/sites/cccrg.cochrane.org/files/public/uploads/how_to_grade_revising_1_december_2016.pdf

Hello! I completed and published this review effectively using the NHLBI checklists to assess risk of bias. They're easy to use, specific to study design, reviewers liked them, and they provide a very detailed view of the risk of bias. My suggestion would be to take care when talking about quality vs. risk of bias with these checklists, because the literature provided by NHLBI is very clear about using it for the latter over the former. I also used them more as a commentary on the state of the body of literature that I was evaluating as a whole, rather than singling out studies with high risk of bias and talking about them individually. 

Best of luck! 

Kindly reach out to Dr. John Ng'asike of Mount Kenya University, Early Childhood Studies Department , Thika Main Campus for assistance. He is a specialist.

There has been quite a lot of research on how to construct survey items, if that is what you want. Here is a review by two experts in that field.

Hi there

I have used it for cross-sectional, using a modified version. See this link http://www.biomedcentral.com/content/supplementary/1471-2458-13-154-s3.doc&rct=j&frm=1&q=&esrc=s&sa=U&ved=0ahUKEwiN5OL2suTLAhUj2qYKHVzdDJwQFggZMAA&usg=AFQjCNF4cUzs2QLQMrFnz9bu6AnTWpokog

Ride quality is subjective. There is no 'correct' way to measure it. Hence, different groups of people have come up with different ways of defining ride quality.

Neuropsychol Rehabil. 2016;26(1):1-35. doi: 10.1080/09602011.2014.977924. Epub 2014 Nov 10.

Impact of rehabilitation on self-concept following traumatic brain injury: An exploratory systematic review of intervention methodology and efficacy.

Ownsworth T1, Haslam C2.

Author information

1a School of Applied Psychology and Griffith Health Institute , Griffith University , Mt Gravatt , Australia.

2b School of Psychology , The University of Queensland , St Lucia , Australia.

Abstract

To date, reviews of rehabilitation efficacy after traumatic brain injury (TBI) have overlooked the impact on sense of self, focusing instead on functional impairment and psychological distress. The present review sought to address this gap by critically appraising the methodology and efficacy of intervention studies that assess changes in self-concept. A systematic search of PsycINFO, Medline, CINAHL and PubMed was conducted from inception to September 2013 to identify studies reporting pre- and post-intervention changes on validated measures of self-esteem or self-concept in adults with TBI. Methodological quality of randomised controlled trials (RCTs) was examined using the Physiotherapy Evidence Database (PEDro) scale. A total of 17 studies (10 RCTs, 4 non-RCT group studies, 3 case studies) was identified, which examined the impact of psychotherapy, family-based support, cognitive rehabilitation or activity-based interventions on self-concept. The findings on the efficacy of these interventions were mixed, with only 10 studies showing some evidence of improvement in self-concept based on within-group or pre-post comparisons. Such findings highlight the need for greater focus on the impact of rehabilitation on self-understanding with improved assessment and intervention methodology. We draw upon theories of identity reconstruction and highlight implications for the design and evaluation of identity-oriented interventions that can supplement existing rehabilitation programmes for people with TBI.

Work. 2012;41 Suppl 1:4782-9. doi: 10.3233/WOR-2012-0764-4782.

Comparison of randomized and non-randomized controlled trials evidence regarding the effectiveness of workplace exercise on musculoskeletal pain control.

Moreira RF1, Foltran FA, Albuquerque-Sendín F, Mancini MC, Coury HJ.

Author information

1Physical Therapy Department, Federal University of São Carlos, Rodovia Washington Luis, Km 235. Zip Code 13565-905, São Carlos, São Paulo, Brazil. roberta.carreira@gmail.com

Abstract

Evidence synthesized based on randomized controlled trials (RCT) results are recognized as the pinnacle of research excellence; however, the conduction of RCT in workplace environment is not always possible. This study comparatively reviewed evidence from RCT and non-RCT studies in which participants performed workplace exercise for musculoskeletal pain control. Up to February 2011, PubMed, MEDLINE, Embase, Cochrane, PEDro and Web of Science databases were searched. All trials that evaluated workplace exercise interventions for controlling musculoskeletal pain were included. The PEDro scale was used to rate the studies' quality, PRISMA and Cochrane recommendations were applied, and association between frequencies of effect size categories (small, moderate, large) from various outcomes by study type was tested (2x3 contingency table). The search yielded 10239 references in English, from which 21 RCT and 12 non-RCT were selected. Both groups of studies presented methodological flaws including descriptions of randomization, blinding of examiners and absence of intention-to-treat analysis for the RCT, and further absence of controls and blind assessor for the non-RCTs. RCTs had significantly more moderate and large effect size reported in their results compared to non-RCTs (p=0.04). Considering the difficulties in randomizing participants in occupational settings, all studies would benefit from better describing pertinent methodological information.

You can use different tools in your experiment. For example : 

So, you can define an experimental pilot protocol and testing with users. 

 CMMI and ITIL are currently used for assessing design and delivery of software services. Combining the two would be aligned with ISO 9001 standards applied to software environment. Prior to that I actually liked old ISO document for software development that I really liked. Actually, some aspects of that document have been implemented in the most recent ISO 9001:2015 issue.

Other methods include Malcolm Baldrige Award guidelines or its national version in different countries. 

I use my Six Sigma Business Scorecard framework to assess the business performance.

I also use McKinsey's 7-S methodology to assess an organization's overall capabilities. 

Thus, assessing processes using ISO, performance using Business Scorecard, and Culture using McKinsey's 7-S gives a very good perspective of a business. I hope it helps.

Thanks,

Praveen

Dear Hassani

Thank you for your reply. Could you share those article(anyone) to me please ? 

Many thanks.

The NHLBI tool is an excellent tool that is being widely used for this purpose. 

Check the below link to access the full tool. There is special guidance with regard to retro studies

Best of luck

M

Software measurement will be more accurate. Try using ImageJ free software. Just calibrate the software with a stage micrometer that you have taken a digital image of at the same magnification used with your testicular tissue sections. Any further call for help will be welcome!

It's helpful if you can do a scoping exercise to get a feel for what's out there. Usually you'd have to do this when working up the systematic review protocol. As already mentioned above cross-sectional studies can't indicate predictive risk, only an association. If there are sufficient RCTs then you might justify focusing only on RCTs but if not then it would make sense to include other longitudinal designs. If you do include multiple study designs then you should use different critical appraisal tools to cover these. Downs and Black and Newcastle-Ottawa as already mentioned above are useful but not without limitations. When selecting a critical appraisal tool it may help to pilot test it on a subset of studies (e.g. some identified in scoping) as this may identify whether it needs modifying for your particular question and can sometimes highlight issues not foreseen. Deeks and colleagues reviewed 194 different tools for critically appraising non-randomised studies, including those mentioned already (Health Technology Assessment 2007; 36: 666-676). Although a fairly old paper now I think this is still useful as a general guide due to the range of tools covered. New tools are continually being developed, e.g. the Cochrane Collaboration has developed "ACROBAT-NRSI" for non-randomised studies although it is a beta version at present and in my opinion is quite a "clunky" tool to use because it is fairly complex (I think some general principles underpinning that tool are really useful  though - but it is probably too early to use this approach since more testing and validation would be needed). Of course, there is considerable flexibility with any tool to modify it so long as you clearly report how and why this was done. It is important to justify why you used any particular tool(s) and to be honest about any limitations, noting this in the systematic review report.

Regarding the outcome measures to include in searches, you might find that you don't need to specify this level of detail in the search strategy - a pilot test of your draft strategy to see whether it picks up the outcomes you need can be very valuable. If you were to find large numbers of bibliographic records which don't report relevant statistical outcomes then perhaps you could try adding statistical terms in to the search strategy to limit the searches, but you would have to be confident that you wouldn't be excluding studies that don't report statistical outcomes in their title/abstract (or whatever fields you are searching).

Hope your review goes well... Good Luck!

Prisma or maybe, CASP?

I emailed Prof Nick Black, co-author of the checklist (Nick.Black@lshtm.ac.uk) , who sent back the following reply below.

This aspect of the questionnaire would not be for the feint hearted, and I can't see how anyone who is doing this is also complying with their stated "15-20 minutes per article" as time to score.

Anyway, here's the explanation provided by Prof Black: 

Downs and Black checklist (JECH 1998;52:377-384)

Question 27: Power

This is in essence similar to a power calculation.

Decide on what constitutes a clinically or socially significant difference between the two groups being compared  (eg difference in desired outcome 60% versus 50% success)

 

Select a probability value for such a difference – we suggest 5% as commonly accepted value.

 

Select a range of study powers against which you want to assess papers. These are represented as A to F in Question 27. For example, A=70%, B=80%, C=85%, D=90%, E=95%, F=99%.

 

You can now determine the number of subjects that would need to be in the smallest group (though the likelihood is there will be the same number in all groups in the study in question). These are designated as n1 to n8. These can be derived from standard software for calculating sample sizes for randomised trials.

 

Now you can use Question 27 to assess the power of all the studies being assessed by applying the number of subjects in the smallest group to the table and the right-hand column gives you the value (from 0 to 5).

 

Warning: this approach may overestimate the power of non-randomised trials (prospective cohort studies) but there is no simple, alternative method available at present.

The article is relevant to quality: Quality Assurance (QA) is a way of preventing mistakes or defects in manufactured products and avoiding problems when delivering solutions or services to customers; which ISO 9000 defines as "part of quality management focused on providing confidence that quality requirements will be fulfilled".

The phenolic group of tyrosine and tryptophan residues (amino acid) in a protein will produce a blue purple color complex , with maximum absorption in the region of 660 nm wavelength, with Folin- Ciocalteau reagent which consists of sodium tungstate molybdate and phosphate. Thus the intensity of color depends on the amount of these aromatic amino acids present and will thus vary for different proteins. Most proteins estimation techniques use Bovine Serum Albumin (BSA) universally as a standard protein, because of its low cost, high purity and ready availability. The method is sensitive down to about 10 µg/ml and is probably the most widely used protein assay despite its being only a relative method , subject to interference from Tris buffer, EDTA, nonionic and cationic detergents, carbohydrate, lipids and some salts. The incubation time is very critical for a reproducible assay. The reaction is also dependent on pH and a working range of pH 9 to 10.5 is essential.

 Egg albumin was first isolated through successive salt precipitations in 1889, and the procedure was later improved in the first part of the 20th century. The standard purification procedures are labor intensive and difficult to mechanize, which meant that large-scale production of pure Egg albumin was not feasible. With the development of a purification procedure using chromatography techniques, production of extremely pure Egg albumin in commercial volumes is possible.

Egg albumin is very similar in amino acid content to bovine serum albumin (BSA) and can be an excellent substitute for BSA. The tyrosine and tryptophan in BSA are 5.1 g/100g  and 0.6 g/100g of protein, while in Egg albumin are 3.9 g/100 g and 1.2 g/100 g of protein, respectively, hence the concentration range may be similar to each other.

@Pieter - an interesting approach. But this deviates slightly from the Cochrane's recommendation of not producing "scores" for quality assessment. I personally struggle with the Cochrane's approach a bit, because I do find, that - as GRADE point out - sometimes only aspects of a study requires you to down-rate it, and this then results in a study, that may be "ok-ish" evidence-wise to be put into a "high risk of bias" category, whereas things may not be as bad as that summary assessment then suggests.

Seeing that you seem steer away from the Cochrane approach, do you have any experience with how this is received by the epi-community?

Nguyen - in general try to follow the advice provided in the Cochrane Handbook on Meta-Analyses (just google the term) regarding assessment of study quality. Then proceed as the others propose above.

The GRADE scoring system is a good approach to start with. Sometimes you need to adapt certain aspects of your quality assessment however, depedning on what specific question you're trying to answer.

Dear Ramesh,

there are various publications, which deal with the compilation / development of systematic reviews. For your important question, I've chosen the following book:

Centre for Reviews and Dissemination, University of York (2009): Systematic Reviews: CRD's guidance for undertaking systematic reviews in health care. see: http://www.york.ac.uk/inst/crd/index_guidance.htm

You can buy this publication; or there's a free download available, see: http://www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf

According to this publication,"risk of bias assessment" is described as a necessary part of quality assessment (see chapter, 1.3.4, pp. 33-44). Performing quality assessment comprises the consideration of several factors like (see p. 33):

• Appropriateness of study design to the research objective

• Risk of bias

• Other issues related to study quality

• Choice of outcome measure

• Statistical issues

• Quality of reporting

• Quality of the intervention

• Generalisability

So the "risk of bias assessment" is just one part of this important process step, when performing systematic reviews.

The authors of the following publication describe the relation / connection between these two terms in the same way:

Shea, B.J., Grimshaw, J.M., Wells, G.A., Boers, M., Andersson, N., Hamel, C., & Bouter, L.M. (2007): Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7(10). see the following links:

Perhaps you are interested also in the following publication, which deals with the topic "risk of bias" extensively:

Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters LM, Santaguida PL, Shamliyan T, Singh K, Tsertsvadze A, Treadwell JR. (2012): Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions. Edited by: Agency for Healthcare Research and Quality Methods Guide for Comparative Effectiveness Reviews. AHRQ Publication No. 12-EHC047-EF. See the following links:

Good luck with your research project. Kind regards, Detlef

While I agree that papers providing only a matrix-free calibration plot are virtually useless as far as real-world sensing is concerned and should not be published, I believe that many researchers are not interested in commercializing their devices, but are more interested in advancing the state of the art by developing novel approaches to solving problems.  In these cases, the technology is then available for those in the sensor industry to develop into marketable devices.  Not everyone is an entrepreneur who is interested in commercialization, but this should not preclude them from doing good analytical research.

If I remember correctly the graph of ExploreDTI shows deviations of single volumes (or slices) from the remaining set. It should be an intuitive and quick graph indicator. However, rather than relying only on graphs produced by software, I would suggest visually inspecting all the volumes. All you need to do  is  find a software that can open 4D files and loop through them quickly. MRIcron cannot do this. FSL have a couple of +/- buttons to loop through 4D frames. I used DTIstudio for visual inspection. In DTIstudio the volumes are listed in a dropdown menu on the right. Click once this ara and use up and down arrows to loop through volumes. It gives also an idea of subject motion, because the quick motion of volumes is perfect for the eye to detect motion. It doesn't take more than 30 seconds to have a sense of whats going on with the dataset. You can also see stripes or lines in single volumes, which may indicate a single corrupt slice.

My advise is to remove volumes that are entirely corrupt. Some say 2-3 volumes out of 32 is the limit; of course your DTI deteriorates with gradually more orientations removed. It is also important at this point to make sure different groups of subjects do not have a different level of image corruption. You may find out differences in FA that are in fact related to the difference in number of orientations left after cleanup (yes, the number of orientations seriously impacts FA). The degree to which this is true may vary, depending if you remove 5 orientations from a dataset with 60 or with 22 orientations. It may depend also whether those 5 are randomly distributed or fall on the  same angulation.

Hope this helps.

I am not sure what is intended to be "gathering survey/questionnaire data".  Surveys, population surveys, should follow some high qualities of sampling (well description of sampling frame, steps on sampling) which needs to be followed by a good statistical analysis considering stratifications, clusters, weights, and also missing data to adequately adjust to represent the population. More than a check list it is important that who is performing the systematic review have an extremely domain of methods and statistics, so the person can really judge if the study was well set.

I am skeptical of many systematic reviews when I known that who is performing them had not strong knowledge of methods and statistical analysis, or sometimes had no good background about the subject being studied and just follow protocols and check lists. Studying check lists and doing a week  or month of Cochrane courses will not make a person an expert in methods and analysis. It takes a lot a lot of effort to get to it. 

How to evaluate a survey when the person has never studied  in depth surveys and statistical analysis of survey research? 

Thanks. Can you please suggest a portable meteorological parameter measuring instrument that I can carry on a bike.

It will be help full if you upload some sample images

Yes. I think they can be measured in terms of:

a) Student satisfaction,

b) Faculty satisfaction,

c) Impact on society,

d) Knowledge generated

e) Technology transfers.

Look at Deming's PDSA cycle (do Google search) and I think you will get a very different impression than you have from you understanding of PDCA.

PDSA is rapid, rigourous and includes both inductive and deductive learning. It can be the fastest way to learn your way to improvement regardless of context and resources.

So please study PDSA a little more before drawing your conclusions.

Thank you for you answer.

You might be already familiar with this dated review that contains a number of useful resources compiled by a group from Cambridge from Intl J Epidemiology (2007):

"Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography"

Full text at:

In one sense it would be disappointing if ResearchGate became another mechanism for measuring performance. Once this happens I can see that a lot of the authenticity of ResearchGate will disappear.

I think STROBE mainly looks at the quality of reporting and at best is a surrogate for methodological quality.

Both studies mentioned above do not look at assessing methodological quality of observational or cross-sectional studies for the purpose of a systematic review.

Moja et. al. reviewed the type of methodological quality in a number of systematic reviews but looking at the article a distinction between observational and interventional studies is not found (probably the reason for Jadad score being most common tool).

Seo et. al. simply looked at methodological quality of meta-analyses and systematic reviews so is not pertinent to the current question.

Sanderson et. al. as mentioned by Ian highlights the lack of a rigorous and validated tool for assessment of observational studies.

I find the 'Risk of Bias' assessment tool provided in Review Manager (Available free from Cochrane Collaboration website) the most practical tool for assessment of studies, especially if there are some observational studies in addition to RCTs. This tool simply assesses different aspects of a study for potential bias. The following aspects are scored..

- Allocation (selection bias)  

- Blinding (performance bias and detection bias)  

- Incomplete outcome data (attrition bias)  

- Selective reporting (reporting bias)  

- Other potential sources of bias

It is important to note that no software will be able to find out Bias in a study. The researcher has to develop skills to identify potential sources of bias in the article.

kind regards, raza

While I'm inclined to sympathize with Afaq's point that the debate may not be conclusive, I believe the topic is worth discussing even if it only affects the behavior of individual reviewers and submitters as they see points of view they may not have previously appreciated.

With regard to the sub-points of the original question:

- I have gotten benefit, in one case great great benefit, even from destructive and occasionally insulting reviews. If the reviewer becomes emotional, I can learn what triggered it and perhaps I may learn more about the underlying beliefs and assumptions that are driving thought in a field. If the reviewer sarcastically dismisses an approach, then I had better either justify the approach more carefully, or find a new one. In one case I did find a new one and an important paper resulted. It is important not to over react. At least the journal editor called for a review. That's first base, and not easy to get on! And at least the reviewer took the time to say something. The worst case is getting told "we have many good submissions and can't get to yours." Then you know you didn't make the cut but get no information. A few journals, even IEEE journals, do not give review feedback because the community is small and there is no way to keep the reviews anonymous. In that case it is essential to write the editor or someone else "in the know" and find out what the objection was.

- Timeliness seems to be getting better in the last few years. A delay means either the editor is having trouble finding a reviewer, or has lost track of your manuscript. A follow up note is advisable after 3 months to make sure it is not the latter. That happened to me once. I had waited nearly a year! In another case, an editor asked me to remind him in about 3 months, and I had to do so 3 times (9 months) as he went through 3 reviewers before he found one who was able to complete the review.

- Most reviewers I've encountered are at least civilized in language, but if they don't like your work of course it comes through between the lines.

- The level of understand is the most difficult area, when new ideas or methods are used in a paper. Papers that make incremental improvements in accepted theory or methods are easier to review and get published more quickly. There is nothing to do here except keep trying to find ways to explain the transition from established thinking to your new methods.

Hope this is helpful to somebody. In addition to publishing in my original field, I have published cross-discipline in two other fields and do not have a PhD in any of them. So believe me, I've seen just about every possible response from a reviewer (including good ones). But it is possible to get through if you keep polishing your work and are willing to change it when you realize a better idea.

Dear Basem

The question you asked is highly dependent on the type of the included studies.

For diagnostic, therapeutic, prognostic and virtually any kind of studies we have numerous checklists.

What I can recommend is to use the checklists not the scales (which give a number for quality of each study).

I personally use Oxford center for evidence based medicine checklists

http://www.cebm.net/?o=1040

Ramin

Unfortunately I can say that I have been through a 'quality management' process in an Australian university, and in my view the process had poor outcomes. Defining the students as clients was a bad idea in terms of educational outcomes. This has shifted the focus in the university from being quality oriented in terms of educational outcomes to being customer friendly and therefore reducing standards to satisfy the 'client'. Universities have to goals, in particular in engineering: 1. to be a place that delivers knowledge and teaches the student how to tackle real-life professional problems 2. to deliver graduates that are suitable for the modern industry. With these goals in mind, the student is the raw material that is turned into a product and the real customer is the society into which the product is delivered. Shifting the customer focus to the students has been, as I said, a very bad move. The majority of students seek to obtain a degree in the easiest possible way, so they seek to obtain the minimum level required rather than the optimum level. In short, unfortunately I have to say that quality standards can be introduced, but not by defining the student as a client, because if the academic is just a service provider, then the academic outcomes have to decline if the university is also not very selective in admissions.