Data Integration - Science topic

To integrate and analyze multiple microarray datasets, including specific phenotypes and control groups, there are several tools and approaches available. Here is a general outline of the steps you can follow using R packages:

Data preprocessing and normalization:

-Download the microarray datasets from the GEO database.

-Import the datasets into R using packages such as GEOquery or limma.

-Preprocess the raw data by performing quality control checks, background correction, and normalization using methods like RMA (Robust Multi-array Average) or quantile normalization.

Sample selection:

-dentify the specific phenotypes you want to include in your analysis and select the corresponding samples from each dataset.

-Ensure that the control groups are properly matched or selected from the datasets.

Data integration:

-Merge the selected samples from each dataset into a single integrated dataset.

-Perform batch effect correction if necessary to address any systematic variations introduced by different datasets using methods like ComBat or SVA.

Differential expression analysis:

-Use statistical methods such as linear models or empirical Bayes methods to identify genes that are differentially expressed between the selected phenotypes and control groups.

-Perform appropriate statistical tests, such as t-tests, ANOVA, or regression models, depending on the experimental design and research question.

Functional enrichment analysis:

-Once you have a list of differentially expressed genes, perform functional enrichment analysis using tools like clusterProfiler, enrichR, or DAVID to identify overrepresented gene ontology (GO) terms, pathways, or biological functions associated with the gene list.

Visualization and interpretation:

-Generate informative visualizations, such as heatmaps, volcano plots, or pathway enrichment plots, to visualize the results.

-Interpret the findings based on the biological knowledge and existing literature.

Some R packages that can be useful for these steps include GEOquery, limma, sva, clusterProfiler, enrichR, ggplot2, among others.

Dear Mouncef Naji ,

Look the link, maybe useful.

Regards,

Shafagat

Galaxy-P (http://galaxyp.org/) might be useful.

There are various ways to integrate different datasets. You may extract and generate common feature from the dataset and integrate the datasets in similar fashion as people do in relational or object oriented databases. There is need to be specific over dataset.

Dear Mahdis Dezfouli,

Data fusion is the process of getting data from multiple sources in order to build more sophisticated models and understand more about a project. It often means getting combined data on a single subject and combining it for central analysis.

or

Data fusion frequently involves “fusing” data at different abstraction levels and differing levels of uncertainty to support a more narrow set of application workloads.

or

Data fusion is the process of integrating multiple data sources to produce more consistent, accurate, and useful information than that provided by any individual data source.

Various types of data fusion work in different ways:

Low, intermediate and high-level data fusion – and likewise distinguish geospatial types of data fusion from other types of data fusion. Another specific type of data fusion is called “sensor fusion” where data from diverse sensors are combined into one data-rich image or analysis.

Data fusion is broadly applied to technologies, for instance, in a research project, scientists might use data fusion to combine physical tracking data with environmental data, or in a customer dashboard, marketers might combine client identifier data with purchase history and other data collected at brick-and-mortar store locations to build a better profile.

Data fusion involves a level of concrete definition from something called the Joint Directors of Laboratories Data Fusion Group which produces six levels for a data fusion information group model:

02. Data integration :

Data integration is a process in which heterogeneous data is retrieved and combined as an incorporated form and structure. Data integration allows different data types (such as data sets, documents and tables) to be merged by users, organizations and applications, for use as personal or business processes and/or functions.

or

Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. A complete data integration solution delivers trusted data from various sources.

or

Data integration involves combining data from several disparate sources, which are stored using various technologies and provide a unified view of the data. Data integration becomes increasingly important in cases of merging systems of two companies or consolidating applications within one company to provide a unified view of the company's data assets. The later initiative is often called a data warehouse.

or

Data integration in the purest sense is about carefully and methodically blending data from different sources, making it more useful and valuable than it was before. IBM provides a strong definition, stating “Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information.”

An example of data integration in a smaller paradigm is spreadsheet integration in a Microsoft Word document.

I hope I have answered your question.

With Best Wishes,

Samir G. Pandya.

Sorry, I am not familiar with and experized in the identification of similar individuals in the OBDI. Maybe some other guys could help you.

Best,

Yanshi

It is very important to maintain a high level of cyber security also in the area of computerized health and social care institutions.

Unfortunately, despite the assurances of companies that run social media portals, the information contained on these websites is not always fully secured against the activities of cybercriminals.

In addition, the issue of downloading data from social media portals by large companies to Big Data database systems should be added in order to process them for marketing purposes.

The issue of privacy in social media is very important and is related to the security of personal information. Privacy is at risk in terms of information posted on social media portals.

I invite you to the discussion

why not.it is a good idea.

As the prescription drugs, getting more severe side effects, the low acupuncturist reimbursement makes the alternative treatments more useless.

When there are more symptoms and more severe diseases, the patients need more needles and/or longer treating time. Due to there is no acupuncturist's seat on any insurance company's board, the insurance low payments to the acupuncture clinics makes an acupuncturist need to treat multiple patients in an hour. That destroyed acupuncture treatment effectiveness.

It depends on what the data is being used for!  If it is the pharmaceutical or biological arenas it must be compliant with 21 CFR part 11 (or EU equivalent depending on where you are selling).  Many places also look at GAMP5.

I recommend https://opendatakit.org if you plan to collect information.

Hii, not a recursive sort, in my given example in list [[1]] there are 6 sub-lists like; [[1]][[1]], [[1]][[2]] .... [[1]][[6]]. ans each has different length like 4, 3,6 and so on... So i want to sort this sub-lists [[1]][[1]], [[1]][[2]].... by their given length which is 6 for [[1]][[1]], 4 for [[1]][[2]], and the sub-list which has 0 length would not be considered. Thanks.

Hi Steve,

you can use SSC-H vs SSC-W.

Dear Rishabh,

The following links provide data integrity in pharmaceutical industry:

1-http://www.slideshare.net/skvemula/presentation-on-data-integrity-in-pharmaceutical-industry

2- https://www.parexel.com/files/2614/2184/8648/Schmitt_Regulatory_Handbook_final_Jan_2015.pdf

3-http://www.ey.com/Publication/vwLUAssets/ey-data-integrity-compliance-in-the-pharma-industry/$FILE/ey-data-integrity-compliance-in-the-pharma-industry.pdf

4-http://www.pharmtech.com/data-integrity-key-gmp-compliance-0

5-http://www.pharmamanufacturing.com/articles/2011/087/

6-http://www.in-pharmatechnologist.com/Regulatory-Safety/Around-80-of-data-integrity-issues-QC-related-says-CPhI-expert

7-http://www.ivtnetwork.com/article/data-integrity-issues-pharmaceutical-companies-part-2

Hoping this will be helpful,

Rafik

Please, go through the following links

As a modeller I found data on soil carbon, its changes with time, soil bulk density are lacking.

We usually get help from long term fertilizer trials datasets; unfortunately these are inadequate and heavily skewed towards crop data.

Soil data are few and far between.

Regards

Thank you Kotosz. 

This seems to me to be suitable for a multilevel model, where you have periods nested within individuals, nested within groups. I am not sure why you need the year if you have periods? Also, I am not clear on what sort of post-hoc test you are looking for? What is it that you want to analyze as a post-hoc analysis?

Let me take a different approach to your question on synthesizing. Synthesis is about writing, putting together themes to establish an interconnected and organized point of view using information from multiple sources, which can include a meta-analysis. By synthesizing,you are trying to create a new understanding, and I do agree that the best way to do this is through a systematic review, which can be quantitative or qualitative. It depends upon your question.. You need to be able to write effectively to do this. Here are some sources that may help:

There is very good research on matching patterns by Scharffe and Euzenat. I have also done some work be cataloging and defining typical ambiguities/mismatches (see attached papers).

you can independentize your datasets as several datasets (eventually grouping them by eta values), then declare your ranges independently on the datasets

One error on 60% of surveys does not constitute a 60% error rate -- the rate would be dependent on # of responses entered.  Actually, only l error on 60 out of 100 surveys doesn't sounds unusual.  All manual data entry is subject to error. The best way is to double enter, as you are doing. If you could load the survey on to a telephone survey software, you would reduce your error rate and lessen the # hours in data entry.

use origin 8.5 it may be useful to you, for this should have raw data (excel sheet) 

It's also good to work into the direction of the TPC (http://www.tpc.org/). It's widely know, acceptable benchmarking practice with very robust parts.

In the paper recommend by the Neamat El-Tazi "OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases" they are also considering part of the TPC called TPC-C.

P.S. Many thanks to the Neamat El-Tazi for recommending "OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases".

3Omics is also good. But, even though one uses, same material for both studies, it would be hard to compare for obvious reasons. A transcriptome can easily come close to show expression of 99% of the coding/ expressing regions of the genome, while identifiable metabolomes is only 5-10% of the genome with best possible available tools, annotations and mass-specs. thus limitation is "annotation" and the "diverse class of chemistry" of metabolome in general. Proteomes lay somewhere in the middle of both. Combing both may help you 'focus on few pathways' where most of your metabolites concentrate (instrumental and chemical bias/ basis being the major reasons!), but even would be hard to explain why in a metabolic pathway, one molecule upstream is upregulated, the one below it is downregulated while many do not show any change. Statistics alone would not help in deciphering the 'missing dots' in metabolomic datasets.

thank you sir, right now  this facility not available in our institute we have send the sample to other institute   they analyzed the sample according to our instruction.   they operate the instrument   in negative mode.

Four pictures, just to share with you the basic idea of evolution of social relations within a virtual environment [bit it works also within real organizations].

—g

Hi Cliff, a problem with making such a simulated dataset is that there not so much known on how you should introduce "noise" in a realistic fashion. Most researchers agree that the size of a cluster of duplicates is Zipf distributed and some have proposed some models for introducing typographical errors, but apart from that, it would be "guessing" what a realistic error model would be. For example, abbreviations, multi-valued attributes, subjectiveness (e.g. the musical genre of a CD)... And of course, if there is no realistic error model, simulating a dataset tends to be biased to work good on the algorithm you want to test.

It's very hard to answer this question in general without taking into considerations what your specific needs are. Also, "Data Warehouse" is a pretty general term which basically can mean any kind of technology where you put in your data for later analysis. It can be a classical SQL database, Hadoop (yes, Hadoop can be a Data Warehouse, too), or anything else. Hadoop is a general Map Reduce framework you can also use for a lot of different tasks, including Data Warehousing, but also many other things. You also have to bear in mind that Hadoop itself is a piece of infrastructure which will require a significant amount of coding on your part to do anything useful. You might want to look into projects like Pig or Hive which build on Hadoop and provide a higher level query language to actually do something with your data.

Ultimately you have to ask yourself what existing infrastructure is already in place, how much data you have, what the kind of questions are you want to extract from your data and so on, and then use something which fits your needs.

I would go for the semi-supervised learning algorithms for helping the data integration. Also, clustering techniques can probably work for the visualization.

I've worked with geographical datasets for many years and have worked on similar problems. It helps a little to think of the problem as a nearest neighbor problem in 2D space. That brings together possible matches, but the crux of the problem is how to measure the similarity between two geographic features, that is, how to compute a distance function for each pair. For text similarity, there is the Levenshtein edit distance. I have usually improved on the basic algorithm. For comparing the geometry of 2 features, I have used dynamic programming algorithms. Levenshtein edit distance is itself computed by a dynamic programming algorithm, and the algorithm I have used for geometries is somewhat analogous. But those only deal with some of the complexity of integrating two geographic datasets. It is not simple.

Abstract:

Thanks for great introduced papers:

1- there are horizontal and vertical integration for each OMICs data: e.g. Breast cancer

a) integration of Breast cancer microarrays (horizontal),

b) integration of Breast cancer Protemoics (horizontal),

c) integration of Breast cancer GWAS data (horizontal),

finally:

d) integrating integrated data of proteomics, genomics and transcriptomics (vertical)

Mta-analysis methods for horizontal and vertical combining of data:

1) Vote counting, 2) Combining ranks, 3) Combining p-values, and 4) combining effect sizes ???????? I need a more detailed papers about these analytic methods. Are methods eaquel for a, b, c, and d (above)?

Then I have learned that there are several ways to analyze pathways including:

1) Overrepresentation methods (EASE, GOStat, DAVID, MetaCore)

2) Gene set enrichment methods (GO, GSEA, SAFE)

3) Set-based methods (PINK)

4) Modeling methods

5- Network-based methods (KEGG, GSEA, PANTHER, DAVID, GO, MSIGDB)

6- Text mining methods

Right?

Have you tried semanticSBML (http://semanticsbml.org/semanticSBML/simple/index)? The problem you mentioned, merging systems biology models, is a tough one and a mostly unsolved one, so it will most likely include lots of manual work.

Depending of the organism that you are looking for. Generally the database with more reliable curation are associated with single organism-like databases, for example SGD (http://www.yeastgenome.org/) for Sacharomyces, FlyBase (http://flybase.org/) for drosophila, Wormbase (http://www.wormbase.org/) for C.elegans or TAIR (http://www.arabidopsis.org/) for Arabidopsis and Genome Reference Consortium at NCBI (http://www.ncbi.nlm.nih.gov/projects/genome/assembly/grc/) for human, mouse and zebrafish. Beyond these databases with intensive manual data curation, other good resource for genomic database is EMBLGenomes (http://www.ensemblgenomes.org/) but the quality of the genome depends of the consortium that produced the data. Also it is interesting databases such as SGN (plants, Solanaceae) where you can find a community curation model based in the assignment of Locus editor role to relevant publication authors (http://solgenomics.net/).

About the software to build metabolic networks, I like Pathway Tools (http://bioinformatics.ai.sri.com/ptools/) associated with MetaCyc database, but there are other options such Blast2GO for annotation (http://www.blast2go.com/b2ghome) and KEGG pathways (http://www.genome.jp/kegg/pathway.html) or Mercator (http://mapman.gabipd.org/web/guest/mercator) for annotation and MapMan (http://mapman.gabipd.org/web/guest/mapman).

Yes, it was. I attest to that statement with [quite] a few publications for evidence.

"Biologists know their pathways" is a dangerous proposition. First, it's not about them, biologists and us, presumably something else. One can't handle an advanced tool effectively without being a biologists. Second, some researchers tend to focus on a single gene, single hypothesis, single pathway, which may or may not turn out essential. This focal approach is common recipe to successfully complete the study, make a publication and miss the discovery. It can be balanced by a broader view provided by those expensive advanced tools with all conveniences, bells and whistles. You don't need this unless you are going to spend hours clicking around trying to figure out the way things happen in the experiment, the meaning of expression level changing in all directions and patterns in chaotic shadows. Simple things like grabbing some Cytoscape, routine GO tables and enrichment tests are perfect if you do your job 9 to 5, perform your duties of highly qualified specialist and let them clients, biologists, doctors worry about discoveries. This is how core facilities usually operate. I concord with your point, all those "advanced" tools are not as advanced under the hood as one may think reading their sales booklets. They also don't solve the problems and make discoveries. However, these fancy tools give you a better fighting chance if you know how to use them.

I suppose you refer to the optimisation problem of finding a linear combination of p vectors of length n that have maximum variance (or sum of squares when directly linking it with SVD) ..and then adding up an uncorrelated constraints for the next linear combination (having maximum variance but being uncorrelated with the first one). This usually leads to to eigen-decomposition of the Xt(X) (with your notation).

You may also have a look at the bilinear optimisation, equation (5) in the following paper, expressed using a tensor formalism.

Leibovici, D.G. (2010) "Spatio-temporal Multiway Decomposition using Principal Tensor Analysis on k-modes: the R package PTAk." Journal of Statistical Software, 34(10), 1-34. I think the paper is uploaded on my ResearchGate.

didier