Data Quality - Science topic

Maybe you can use taxonomy files from PR2? It is regularly updated and there is an R package: https://pr2database.github.io/pr2database/index.html

The plot shows that the average quality per base alongside your 150 pb reads is very high which is good. This result is kind of normal when the sequencing has been outsourced. Most companies will give you prefiltered fastq files containing only reads with high quality. You can ask your sequencing provider if that was the case, although sometimes you can also find that info in the report they send together with your fastq files.

In case-control studies, you can do it in multiple ways. I assume you already know how to get the controls. Imagine, you are performing secondary analysis, you have a dataset with 200 patients, here you have 30 people of SLE (case), and 170 people without SLE (control), as we all know matching 1 case with '1 to 4' controls provides adequate power, don't match more than 4 controls for 1 case.

First, match 1:1, that is 1 control for 1 case. As sex is almost always binary, you don't have to manipulate this variable. But age is not binary, so you have to determine the age range you would go. For instance, a 20-year-old man could be matched with 18-25-year-old men in your dataset, this way you could get. Create a categorical variable for age by specifying that range. Also, create a binary variable of who are cases and who are controls. Then, export these 60 people from the dataset and can perform your analysis. If you want to get 1:2, the procedure is the same (export 90 people), but whatever you do, prespecify the hypothesis, sample size, and analysis plan in the beginning. Best of luck!

You can also see propensity score matching or inverse proportional treatment weighting but that procedure is a little complex and you need somebody with advanced statistical analysis knowledge to do that.

Hello,

For the general case of time series anomaly detection, several benchmarks have been recently proposed. These two benchmarks contain labeled time series from different domains (for instance, room occupancy detection from temperature, CO2, light, and humidity [1] or accelerometer sensors of a wearable assistant for Parkinson's disease patients [2]).

You may find the links to the benchmarks below. Good luck with your thesis!

- TSB-UAD: https://github.com/johnpaparrizos/TSB-UAD (paper: )

- TimeEval: https://github.com/HPI-Information-Systems/timeeval-evaluation-paper (paper: )

[1] Luis M. Candanedo and Véronique Feldheim. 2016. Accurate occupancy detection of an office room from light, temperature, humidity, and CO2 measurements using statistical learning models. Energy and Buildings 112 (2016), 28–39. https://doi.org/10.1016/j.enbuild.2015.11.071

[2] Marc Bachlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jerey M. Hausdor, Nir Giladi, and Gerhard Troster. 2010. Wearable Assistant for Parkinson's' Disease Patients With the Freezing of Gait Symptom. IEEE Transactions on Information Technology in Biomedicine 14, 2 (2010), 436–446. https://doi.org/10.1109/TITB. 2009.2036165

You might try deleting the current product system and make new flows, processes and product system. Seems like some mistake has been made in previous steps.

hi,

quantitative techniques for evaluating GAN generator models are listed below.

for more info:

Best wishes..

Developing countries can make use of a lot of health research conducted in developed countries, for these researches are scientifically proven, while those of developing countries are of short data for the universities there do not pay enough to do excellent researches.Regards.

Dear Prof. Dr. Devaraj Acharya , several methods could be used to ensure qualitative research analysis, such as Guba and Lincoln’s Concepts for defining and investigating quality in qualitative research as the following RG link. Kindly visit.

Kind Regards,

Hi Alba.

I had experience with Novogene, both for whole-genome and metagenomics.

The sequencing is very good, and I had no problems at all in terms of quality.

I don´t recommend to subscribe for their bioinformatic analysis, since the analyses performed are pretty basic and can be done easily by the researcher, and many times you need more complex analysis that are not included in the price.

Let me know if you need more information.

Ricardo

Then in your case, I would do a comparative analysis of the variables. For example, I would build a scatterplot between variables on the reference data, and I would also build the same scatterplot between variables in the data under study.

I think you need to analyze the hydrology of the study area. At the same time, I think, there may be mistakes

This paper and its reference list should help:

Marsh, H. W., Martin, A. J., & Jackson, S. (2010). Introducing a short version of the physical self description questionnaire: new strategies, short-form evaluative criteria, and applications of factor analyses. Journal of Sport and Exercise Psychology, 32(4), 438-482.

Yes, you can make 10 ul reaction mixture for taqman! However, during running PCR you need to make sure that you have changed the the total volume to 10ul instead of 20ul.

Best of luck

Hi Johnson Masinde , thank you for your answer. I requested the file to have more details but it seems answering partially. I'm more interested in the data improvement methods in case of limited datasets.

Ecological foot is more suitable representative of environmental degradation, carbon footprint is a small portion.

Collection of real time and crisp data and its validation.

I have a few ideas. Try to find out who installed the equipment and collected the data, who interpreted and edited data for final data set entry. What were their methods and quality control. Do you have just digital data, or examples of strip charts, punch tapes, site visits with comments, etc.? Who worked up data, how did they deal with missing records? What equipment were used, was it installed correctly for research, etc.? Are there documents such as a data library, publications, pictures? Can you visit sites of data collection, and look over any remnants as staff gauges, benchmarks of cross sections or bridge sites, historical aerial photos covering area, streamflow records used for stage discharge curves, adjustments with time. Location of rainfall sites relative to watershed measured and topography. I think you would find in discussing this topic with the US Geology Service professionals and research hydrologists or others involved with this work, a substantial amount of time is used in development of a quality data set. Someone should have documented the specifics, but realizing how sometimes this is not done or personnel changes and may cleanout and dump files or records, even reports, much can unfortunately be lost with time. Some research grade stations like USFS Coweeta Hydrologic Experiment Station or agencies like USGS take much effort in cataloging, filing, logging and storing past records, notes, etc. for posterity.

Also, plot the time series data out. Do the data of the various stations seem to correlate with each other. Are the readings frequent as many times per day, which is a good sign, or infrequent as daily readings, not as good quality. Do the flow hydrographs coincide with rainfall events? Temp changes with day and night, rainfall events, etc. Are there other streamflow data stations in vicinity which have good correlation with records taking into account some time differences in storms and response.

Hi Juelu Wang , you can see the conductivity of your measurements on the Zeta Report or also listed in the Zeta workspace. Yes, it appears that you have enough scattering intensity signal. If your overall mean zeta potential is very close to zero then it may be difficult to improve the data quality. The diffusion barrier method may improve. Here are a few comments that may be useful:

on mean zeta: https://www.materials-talks.com/blog/2017/09/05/what-is-the-zeta-deviation/

on measuring in high salt (just for info): https://www.materials-talks.com/blog/2020/07/15/measuring-zeta-potential-in-saturated-ionic-strengths/

on diffusion barrier: https://www.materials-talks.com/blog/2015/11/24/diffusion-barrier-method-the-practical-details/

and finally, feel free to add a data file: https://www.materials-talks.com/blog/2014/08/19/faq-how-can-i-submit-a-data-file-to-the-help-desk/

Ms Breanne R. Helmers

Unfortunately, I found another recent study on the quality decrease of MTurk data: An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results by Michael Chmielewski, Sarah C. Kucker.

https://journals.sagepub.com/doi/abs/10.1177/1948550619875149?journalCode=sppa

You can apply some basic clustering techniques i.e. k-means, DBSCAN and so on for the outlier detection and removal.

The low quality data may be noisy and untrained as you have written such data could be reprocessed but if data is irrelevant , out of context and not explained the problem and requirement then the data would be futile . The unsupervised method or rule based method could help by the understanding the new problem and the rules those would help to take decisions .

Excellent quality. I have run RNAseq, BSseq and genome sequencing (our own libraries and novogene-built libraries) though their facility and always get fantastic data.

Hadi Fadlallah , yes. That should decrease computational expenses, help to perform an investigation of a subset instead of the whole set. That is similar to a data science process where a small dataset is analyzed afterwards the methods are applied on big data set.

Sorry I'm late to the party but if you know which datum is faulty you should be able to solve the problem. If you don't know how do you decide?. Which came first the chicken or the egg?

David Booth

BTW outliers often get a bad rep but are actually useful.. How do you tell the outliers from the approx. 2.5 % of data that lie in a tail. see the attached

This is like saying a road grader is poor because it doesn't do well on a concrete pavement. It is like saying the fish tastes red, Red is not a property of fish tasting and machine learning is not a form of data.. WORDS HAVE MEANINGS. Thanks to Prof. Arnold Insel for the fish tasting example. David Booth

To address the Data Quality, it should be considered in its totality irrespective of its usage, Once the data is complete it will certainly the serve the purpose no matter if it is for AI, ML or any other application.

To achieve the highest level of Data quality we should try to cover one or all the following dimension of data.

1. Uniqueness

2. Completeness

3. Timeliness

4. Consistency

5. Accuracy

6. Validity

The successful implementation of these dimensions in overall data quality process will decide the ultimate shape of your data.

- Another key point to note is that Data quality is not really a IT challenge to solve and there should be equal participation of Actual Data consumers or Data Creators who are responsible for it. Without complete understanding of data it would be impractical to decide if data is right or wrong.

To implement the Data quality , One should not try to "boil the ocean" and better go with iterative approach.

Step 1: Run Data Profiling to discover the high level data issues.

Step 2: Select the most Critical Data elements(CDE) those are candidate for data quality.

Step 3: Decide the DQ dimension that make sense for given CDE.

Step 4: Define and Develop Data quality rules to check the data error.

Step 5: Produce the exception report(with records details not satisfying the rule criteria) and share it with Data users or Owner

Step 6: Develop process to correct the data issues and monitor the progress.

Step 7: Repeat the Step1 to Step 6 until all the candidate data is covered for data quality process.

Dear Ruslan,

I think it is worth checking out the EasyqpcR (https://www.bioconductor.org/packages/release/bioc/html/EasyqpcR.html). It is is a very comprehensive and well documented package and has been used in a variety of most recent publications. Major advantage is that it can be run via a GUI (using gWidgets; see: https://bioc.ism.ac.jp/packages/release/bioc/vignettes/EasyqpcR/inst/doc/vignette_EasyqpcR.pdf) which makes it very easy to use.

If you are looking for a more advanced tool to analyze your qPCR data which can cope with raw intensity values and perform melting curve analysis etc. I recommend the qpcR package (https://academic.oup.com/bioinformatics/article/24/13/1549/238435) which can do quite a lot of cool stuff.

If you are running a high through put experimental test design maybe check out this tool: HTqPCR (https://academic.oup.com/bioinformatics/article/25/24/3325/235116)

To facilitate qPCR data import there is another nice tool available called ReadqPCR (https://www.bioconductor.org/packages/release/bioc/html/ReadqPCR.html).

For more options check out the comprehensive review from 2014

about qPCR analysis tools which compares among others, 9 R packages. Maybe you will find something more interesting ;) Good luck!

Hello, similar topic discussed here. https://www.researchgate.net/post/Can_anyone_help_me_with_k_fold_cross_validation

Good luck!

Thank you all for the support!!

In some instances we use other softwares to perform the statistical analysis based on the syntax developed and report this in the write-up for data quality and assurance purposes.

Panagiotis Loukopoulos If you're starting with a dry powder you have a fused collection of sub- and post-micron aggregates and agglomerates. Have you measured the SSA of your starch? If so, what is it? The behavior of the material will reflect the bulk size in terms of properties such as flowability, dusting tendency, filter blockage etc. Look at Figure 2 in the attached. Nowhere do you see free, independent, discrete particles < 100 nm. Thus your measurements should be taken on laser diffraction and not DLS. Many starches are also in the 20 micron region for mean size and thus, again, not applicable to DLS. Takse a look at this recent webinar:

May 28th, 2019 Dispersion and nanotechnology

Your Co tube case apparently suffers from low photon flux at the detector (compared to the Cu tube case). Do you have taken the same sample?

It seems to me that you might have got an experimental issue here.

For example the sample height adjustment needs to be improved.

Please play around with the height position of the sample with respect to the x-ray beam (rotation axis of your set up).

Good luck

Sounds like you've created an instrument which is sound as it is based on theory - but this version should only be the pilot version, rather than the final instrument. As you see - there are too many items, which will affect the quality of the data collected. Have you conducted any sort of pilot study - factor analysis - and tested to see how those items relate? You'll probably find some items are redundant, and possibly even some scales ... use EFA to explore how the items load - then as you delete items - the number of items in each scale will decrease - you can then delete scales on the basis of insufficient items.

I find David De Vaus' work on survey design and validation very useful:

https://www.amazon.co.uk/Surveys-Social-Research-Today/dp/0415530180/ref=pd_lpo_sbs_14_t_0?_encoding=UTF8&psc=1&refRID=ER3KY0K00XZC84NB7GHW

Sampling is the basic part of data science . In statistical quality assurance few imperical formule are available of which variance is most important.

Thanks @David -

I take it that it is OK for me to use the data in Appendix 1? Would you happen to have an excel file of those data?

Thanks again - Jim

Political anything can have negative connotation to many people.

Ikhlas Almukahel ,

Thank you for your detailed answer. May I assume that you agree but there need to be more things for data quality assessment?

I think you should refer to standard number 1053. You can find the quality requirement for drinking water

Tea tasting is the process in which a trained taster determines the quality of a particular tea. Due to climatic conditions, topography, manufacturing process, and different clones of the Camellia sinensis plant (tea), the final product may have vastly differing flavours and appearance

The ISIMIP project has a lot data on sustainability:

* Fastq is the sequence + quality information. This file is basically provided by the sequencing company. It is used to control the quality of the reads going in further pipeline. Fastq file is converted to Fasta after adapter trimming and quality assessment.

* Fasta files are the sequence files which have sequence attached to a specific identifier. These are generated after the quality filtering, and size of file is smaller than fastq.

* Qual files are the Fastq minus fasta, this only has quality information and generated after spliting fastq file.

If you are using a pipeline which filters out good quality reads (your threshold), you don't have to think about what fasta and qual file has. Fasta files will become comprehensive after dereplication and clustering the reads into OTUs. Therefore, just follow the protocol. Meanwhile, please go through the manual of mothur to understand how the process works. I suppose you should do it before you start the actual analysis.

CMIN/df is not really used in these large sample studies, so we can safely ignore it. This leaves the outlier of the RMSEA. In my experience, simple models (1 or 2 factors and a low number of items) tend to follow the pattern in your results.

The main difference of RMSEA compared to the CFI (and other measures) is that RMSEA is a measure of absolute fit and does not have any corrections based on model compexity. It appears you may have a very simple model (2 df), although you don't specify here the exact model, so that assumtion be wrong.

Now, the SRMR is also an absolute measure of item fit, but you have a good SRMR. However, according to David A. Kenney (http://davidakenny.net/cm/fit.htm), the SRMR is biased in favor of low df, which you certainly have.

What does this mean? Well overall you have a simple model that captures a good amount of the variance considering how simple it is, but there's still a large absolute error. Do you need to alter you model? Well maybe. If there are clear theoretical reasons to do so, you should. However, considering all fit metrics, it is probably "acceptable," but not "good." So I wouldn't change the model unless there is a good theoretical rationale. Then you'll want to provide a good explanation of the fit metrics and any biases in any article or paper you write based on this model.

So is it a problem? Maybe. Depends on how complex your model really is and if there is a theoretical rationale to update your model.

Is it a problem of data quality? Maybe, but probably not. Most SEM techniques can adapt for variance, missing data and so on. It depends on the estimator you use and the degree of these issues. you don't say what estimator you are using or if the data are continuous, categorical or interval.

Is it because of the low df? Probably, in as much as you probably have a simple model and there are some inherrent biases in some of the measures.

No worries. You're welcome.

If the target labels are in your data, then you have a classification task (not clustering).

At the very early stage, make sure that your dataset is prepared well. Then, try many classification algorithms, where each of them needs to be evaluated with k-fold cross-validation, for instance.

You need sometimes to tweak the classifier based on the nature of the data, such as selecting the right kernel.

Cheers,

Dr. Samer Sarsam

No, unfortunately. I am still working on it. However, take a look at the study attached. In the opposite way, they consider a fixation accurate where the dominant eye is located on the target. Also, what eye tracker are you using? I am using the tobii x3-120 and in their manual they recommend using the dominant eye for more accurate precision. I am finding in my study the dominant eye is more accurate too.

Paterson, K. B., Jordan, T. R., & Kurtev, S. (2009). Binocular fixation disparity in single word displays. Journal of Experimental Psychology: Human Perception and Performance, 35(6), 1961-1968. doi:10.1037/a0016889

The index will be used for classification and for prediction (probably with KNN).

agree with @ Ajit kumar Roy

regards

There are many data analytic tools (usually commercial) that claim that they can provide information about the quality of your files (and some also claim that they can subsequently provide files where the errors are 'corrected'.) The methods are called 'profiling' tools. I am not aware of any that work in a minimally effective manner. Difficult errors are determining 'duplicates' in files using quasi-identifying information such as name, address, date-of-birith, etc. Two records may be duplicates (represent the same person or business) even when the quasi-identifying information as representational or typographical error. Any quantitative information from the two records representing the same entity may have slight (or major) differences. If the records having missing values associated with the data that an individual wishes to analyze, then the missing values should be filled-in with a principled method that preserves joint distributions (e.g., Little and Rubin book on missing data, 2002). The 'corrected' data may also need to satisfying edit constraints (such as a child under 16 cannot be married).

htpps://sites.google.com/site/dinaworkshop2015/invited-speakers

You can find useful tips on relevant papers and books, for example:

Hey Zhou,

I would add, that it critically depends on what you mean by "serious neuroscience & psychophysics experiments".In the linked question, I have collected some related questions to ET use.

In general, if you don't need subject mobility, more restraints are better (head/chin rest, bite bar, etc.) and will yield better results (in terms of accuracy and precision).

In addition, stationary ETs are currently still faster by a large margin. The fastest VOG-based head-mounted ET is around 300 Hz whereas for stationary, head-fixed VOG-ETs you can got up to 2000 Hz. Since head-mounted ETs use a (low frame rate) scene camera to record the participants environment, many systems don't see the need to go to higher sampling rates.

For fixation analyses, which is more common in psychology & related fields, low sampling rates are not a problem.

But as soon as you go for saccades, sampling rate becomes a critical factor.:

Finally, the calibration makes a difference (see Nystroem2013).

TL;DR: Provide more info on what you plan to do. If you analyses saccade peak velocities, get a stationary ET.

Hope that helps, Greetings, David

There are many tools actually. If you find my book chapter useful, it could help you hopefully. Although title says it's basically about RNA-Seq data but you would get idea about many tools that apply to NGS data QC. Below is the link of my chapter.

The RedCap administrator is able to provide access to double entry of data under user rights for your project. You can further define if one or two users will capture the data. At our university it is fairly easy to get the RedCap administrator to make changes to a project.

All the best wit your RedCap project.

Ref:

Dear Christoph Kowalski, it is not exactly what you are asking for (i.e., a review or study reporting on the increased acceptance or satisfaction with the reporting system) but 'timelines' and 'punctuality' in the provision of analytical data is considered as a quality criteria for any monitoring system (even more so in the case of clinical performance analysis systems). 

As an example, the Eurostat Quality Assurance Framework establish both 'timeliness' and 'punctuality in the delivery of metrics' as their 13 principle (page 30, http://ec.europa.eu/eurostat/documents/64157/4372717/Eurostat-QAF-V1-2.pdf) defining 'timeliness' as 'the availability of the information needed to take action at the appropriate moment of decision making', meaning that there has not necessary to be from a precise period, but be ready and available to the professional (i.e., health or otherwise) to facilitate an informed decision. In clinical settings, is established depending on contextual needs. One of the best examples you could use to argue your point is to follow the analogy of the laboratory test in the emergency care as an information system allowing timely decision making based on current data on the patient, thus promoting action to change an urgent situation. This example can be extended to other health information systems already mentioned in the other answers above. 

Similarly, 'punctuality' to deliver those metrics following a predefined delivery plan is also stated as a data quality principle as it sets the expectative of the user of the information system on the data availability within acceptable margins, facilitating the use of the information provided.

Both 'timeliness' and 'punctuality' are quality dimensions that you should take into account when trying to create an information system to be useful in the clinical context. 

don't know.  First I'd want to see the data, and then see what makes sense....  however, since I haven't seen anything... no worries

thanks for replying,

best regards,

joe

It seems useful:

You should always conduct a pilot study / test before going for "big bang" data collection.  The purpose of the pilot study include the following which is not an exhaustive list:

Dear prof. Gorodetsky,

I'd suggest you to analyze the existing literature on your "ill-structured" domain by means of topic modeling techniques. By inferring relationships across topics you can build the dependency model you are after. In the attached paper you can find a methodology we have designed in order to perform such an analysis.

Regards,

Andrea De Mauro

I am afraid that a simple Z - sizer is not an appropriate instrument for such measurements. To be sure in a result of the measurements, they  should be done at 2 or 3 scattering angles and it should be checked that the correlation function processing program allows a repetition of the location of size distribution peaks. We should also note that when particles are large (and their amount is small) in the different times the scattering volume can be provided by different particles. Therefore at each scattering angle few measurements are needed. So for DLS measurements of complex size distribution you must use Malvern DLS sizer with opportunity for angle changing or analogous Nanotrec sizer or Photocor FCN sizer with good program for correlation function expantion on exponents. For Photocor the program DynaLS is sufficient, for other sizers compatible software is needed. Look for the DLS installation of this class.

Yes, I think SEM should help. But a model should enumerate indigenous and exogenous variables with well grounded hyptheses and should not have either type-I and/or type-II errors.

But, it would be necessary to validate with case based inputs.

Fellegi and Holt (JASA 1969) rediscovered one of the fundamental theorems of logic programming.  The mathematics of the models drive the implementations.  Winkler (2003) demonstrated theoretically how edit rules could be logically connected with imputation procedures so that there were guarantees that the joint distributions of fields would be preserved in a principled manner.  The mathematics guarantee that the entire set of rules are logically consistent (including rules on the joint distributions that are standard in statistics).  All Fellegi-Holt-based systems assure logical consistency.

A number of papers such a Galhardas et al. and many papers on variants of symbolic logic assure that rules in languages can be converted to the types of rules (sometimes) in the Fellegi-Holt model.  What the rules-conversion do not assure is the logical consistency of the systems.  We found in small situations that it was very straightforward to convert the English-type rules to suitable tables (sometimes in less than four hours).  In much larger situations, such general rules-based methods would have been very useful.  We never had the resources to pursue the additional features in our production systems.

In some of our systems, we need to convert 100s or 1000s of rules into table format.  The conversion is tedious.  Once in the table-format, it is relatively straightforward (but may take much computer time) to check the logical consistency.

I would use weight of each atomic system, which is a measure that gives size and/or relevance of the system. That's something you have to come up with.

Then simply multiply each weight factor with the respective DQ factor and then use such values to do avg, stddev, etc.

Hi Mohammed

I have a small list of ANT papers which might be interesting for you. I also pulled out a much shorted list of those which address quality issues. I hope that helps.

Regards

Bob

enc

There are many studies linking subjectively obtained, survey-based measures of customer satisfaction with its outcomes. For a summary, see:

Mittal, Vikas and Frennea, Carly, Customer Satisfaction: A Strategic Review and Guidelines for Managers (2010). MSI Fast Forward Series, Marketing Science Institute, Cambridge, MA, 2010.

Hi There

Go to this site: http://istheory.byu.edu/wiki/Main_Page 

It has a huge list of IS Models/Theories in alphabetical order and Actor-Network is there. Click on Actor-Network Theory and scroll down to where is says "IS articles that use the theory" .... there are plenty and some are very recent. Enjoy and Good-luck.

Hello,

For me, possibly the most challenging is/was/will be to identify a niche within a vast amount of knowledge to be able to introduce meaningful research questions that would be novel and could enhance understanding of mechanisms underlying psychological disturbances (Yes, I am a scientist and a psychologist).

Sounds like a philosophical statement. But it matches your question - very broad and philosophical too.

Regards, Witold

DF is improving DQ when (and only when) the different data input streams are to some degree correlated. If not, DF does not make any sense.

Sorry - I cannot give you academic materials because I have none on this topic. The above is from experience and some own work in this area.

http://people.csail.mit.edu/harr/papers/icde2010.pdf.

http://www.neilconway.org/docs/tkde-usher.pdf.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.218.3113&rep=rep1&type=pdf.

There is a direct correlation between source of data, collection of data and error rate when it comes to quality of data. Personally, I think you should pay more attention to how data is being collected because this is the most likely source of data quality breach. The more accuracy you have in data collection, the less the error rate you will observe.

DD=DCM+ER

ER=DD/DCM  

Where

DD=Data Destination (Constant variable- Always known)

DCM=Data Collection Methods

ER=Error Rate

This is an accepted method of sampling in public health and is equivalent to stratified sampling. The sample sizes maybe smaller but allows the development of hypotheses.

WHO does recommend this form of sampling for large studies such immunisation coverage

There are a couple of things going on here that can lead to sparsity.

1.  Incompleteness - this can be dealt with assuming or testing that your data set complies with random sampling.

2.  Censored Data - sparsity could result from the inability (instrumental or otherwise) to measure certain things.  This is often referred to as "Survival Statistics" and you can use that as a search term to find useful information.

3.  An unusual selection function.  Selection functions are very important and are generally unknown at the onset.  They occur depending upon how data was gathered and the (unknown) bias or limitations of that data gathering.  All data samples, in the end, are the result of some selection function convoluted with the intrinsic distribution. This produces the sample distribution 

Thank you for bringing this matter to our attention!. It is still a great help and a real pleasure to read your posts