Data Mining and Knowledge Discovery

Why don't you use a better clustering algorithm. K-means is one of the simplests and also with less accuracy. I might suggest you this paper in order to have a more general idea about the diversity in clustering algorithms http://www.cs.rutgers.edu/~mlittman/courses/lightai03/jain99data.pdf

Some of my colleagues had good results using Correlation-based Feature Selection. Data analysis software supports this technique, e.g. R, NumPy, and Weka.

Rule-Based Semantic Tagging. An Application Undergoing Dictionary Glosses http://arxiv.org/abs/1305.3882

At my work we have a listserve to post questions and thoughts about a very complex health care data set. I find it more useful than a users group meeting. The members of the list are made up of a mix of individuals that have in common a need to access and utilize various portions of the data. The listserve allows a direct answer to a question from people who have been around the data longer and had to deal with the same or similar issues. Basically it keeps us from reinventing the wheel over and over. Just monitoring the questions begins to give on a feel about the depths of the data and best approaches to questions that I need to answer with the data. Often the problem is that there are several locations for the same data like date of death. There are discussions on the reliability of the various locations in the data set. Returning to your question, as for a users group meeting I would assume that there other venues where the results of the data analyses are presented and consider partnering with those meetings to provide a connection point.

Data Mining (DM) is not just a single method or single technique but rather a spectrum of different approaches, which searches for patterns and relationships of data. Data mining is concerned with important aspects related to both database techniques and AI/machine learning mechanisms, and provides an excellent opportunity for exploring the interesting relationship between retrieval and inference/reasoning, a fundamental issue concerning the nature of data mining. As a result, DM is an interdisciplinary filed, the confluence of a set of disciplines, including database system, statistics, machine learning, visualization, and information science. DM is consider the main point in the KDD that include the followin steps Developing, creating, data cleaning and preprocessing, data reduction, choosing the data mining task, choosing the data mining algorithm(s), data mining, interpreting mined patterns and finally consolidating discovered knowledge). Data mining system can be categorized according to various criteria, as follows :
Classification according to the kinds of databases mined
Classification according to the kinds of knowledge mined
Classification according to the kinds of techniques utilized
Classification according to the application adapted While, Information retrieval (IR) or text mining is a field that has been developing in parallel with database systems for many years. Unlike the field of database systems,which has focused on query and transaction processing of structured data, information retrieval is concerned with the organization and retrieval of information from alarge number of text-based documents. as a result, user become need of a tools to compare different documents, rank the importance and relevance of the documents, or find patterns and trends across multiple documents. Thus, text mining has become an increasingly popular and essential theme in data mining. There are many approaches to text mining, which can be classified from different perspectives, based on the inputs taken in the text mining system and the data mining tasks to be performed. In general, the major approaches, based on the kinds of data they take as input, are: (1) the keyword-based approach, where the input is a set of keywords or terms in the documents, (2) the tagging approach, where the input is a set of tags, and (3) the information-extraction approach, which inputs semantic information, such as events, facts, or entities uncovered by information extraction. A simple keyword-based approach may only discover relationships at a relatively shallow level, such as rediscovery of compound nouns (e.g., “database” and “systems”) or co occurring patterns with less significance (e.g., “terrorist” and “explosion”). It may not bring much deep understanding to the text. The tagging approach may rely on tags obtained by manual tagging (which is costly and is unfeasible for large collections of documents) or by some automat.....

While you can tweak the anomaly detection algorithms for better performance, the biggest challenge is in making sure that the data itself is accurate, complete, and consistent.

What if we create a standard for Big Data across multiple disciplines. Can semantic web be more useful that way. I guess that will transfer processing of user query more towards system side then towards user side. This will help inter-discipline data movement which will in turn reduce human effort for creating applications requiring data from different disciplines for experimental results.

I dont know more about cloud compting, but I know about association rule. association rule is usually used to mine data association with each other, how many data combination appear together. in networking it is usually to detect association about user and their activity.

You can use one document classification SDKs such as uClassify. It is a good document classifier project. It is provided as a SDK for a number of languages. Although, what you need is not classification, rather the extraction of data, uClassify can provide this as well in form of keywords (weighted as an addition).

Do you think frcatal distance measurment can solve this problem? Do you have any idea about fractal clustering to predict this dataset?

Mrs Melah, different data means other benchmark data such as iscx dataset while new data refer to own generated data.

I think LDA is the best for this purpose.

You can also find UCI machine learning dataset that is discretized and normalised for frequent itemsets mining in the followin link. http://csc.liv.ac.uk/~frans/KDD/Software/LUCS-KDD-DN/DataSets/dataSets.html The schema definition for the above discretized dataset is available in http://cgi.csc.liv.ac.uk/~frans/KDD/Software/LUCS-KDD DN/exmpleDNnotes.html#chess_KRvK

There exist numerous interestingness measures for association patterns (thus for classification rules). You should start by two research articles : "Selecting the right interestingness measure for association patterns" (KDD 2002) Pang-Ning Tan, Vipin Kumar, Jaideep Srivastava and "Interestingness measures for data mining: a survey" (ACM Computing Surveys 2006) Liqiang Geng, Howard Hamilton. For the specific case of classification rules, you can try your rules in a train/test experiments and see if your rules are as interesting in the training set as in the test set. Another way is to embed the rules in a classifier and see if you obtain good predictive performance.

Thanks to all for your suggestions, and sorry because i was not active for last 1 year in the community....... So unable to respond but its done n i had successfully submitted my thesis on this topic. Thanks & Regards Ankita Gaur

Dear Miguel, raw data processing of satellite gravimetry is quite complicated and needs min. a year to learn for practical use. If you want to mine geoid data in sparsely surveyed regions, it would be better to search for the *results* instead of raw data. Anyway, I do support your decision: to use satellite gravimetry for those regions. It is a common mistake to use high order and rank global geoid models - however they are of high-resolution only in densely surveyed regions. Up to 60th order and rank, these models use 'traditional' satellite gravimetry (perturbation analysis of the orbit elements) - all of the further details came from ground surveys. The up-to-date experiments (GRACE, GOCE, CHAMP) could refine the global geoid up to cca. 150th rank and order. With all of this, if you have just a few local field gravity measurements, it could significantly increase the realiability of your geoid data. Using Stokes is much more easy than processing GOCE ;) Sorry for the length and for being partly off-topic, Gábor

Thanks Oliver and Paul

there are few open source implementations on the web that consider privacy-preserving data mining even though the privacy preserving data mining is an old research topic. This is due to various barriers of adopting privacy-preserving data mining, although there is a big body of theoretical works in this area. for more details about these barriers check [*] anyway, here are some of PPDM implementations GUPT is a privacy preserving data mining platform. GUPT guarantees that the output of data analyses performed using this platform guarantees privacy as defined by differential privacy. https://github.com/prashmohan/GUPT Privacy-Preserving Distributed Data Mining Library (PPDDM-Lib): An open source ANSI C/C++ library of functions and protocols useful to exchange sensible information while computing Data Mining Models from several servers. http://www.di.unipi.it/~atzori/downloads/PPDDM-lib.zip Private-HERMES: A Benchmark Framework for Privacy-Preserving Mobility Data Querying and Mining Methods http://infolab.cs.unipi.gr/pubs/edbt2012/ Differentially private data release for data mining. In Proceedings of the 17th ACM SIGKDD International http://users.encs.concordia.ca/~fung/pub/MCFY11kdd.htm btw, if you are interested in implementing a PPDM technique, I recommend you to check Orange Data Mining open source and Weka 3 Data Mining Software. Both opensources provide excellent development environment to build on. [*] Richard Huebner, Barriers to Adopting Privacy-preserving Data Mining

Respected Sir, We cordially invite you to contribute book chapter for the book "Cloud Infrastructures for Big Data Analytics". The proposed book is to describe the business and technical implications and innovations when there is a seamless and spontaneous synchronization of the two leading computing paradigms (cloud and big data) with the noteworthy advancements and accomplishments in the business analytics domain. Especially how the synergy of these fast-evolving disciplines is to do wonders for the establishment and sustenance of smarter enterprises, a grandiose portion of the smarter planet vision, is to be the major attraction and attention of the book. The chapter topics are found in the following URL. http://www.igi-global.com/publish/call-for-papers/call-details/926 Regards G C Deka

Learning SPARQL by Bob DuCharme has a lot of nice examples on how to use SPARQL aggreagates. http://www.amazon.com/Learning-SPARQL-ebook/dp/B005EI86BS/ref=sr_1_1?s=books&ie=UTF8&qid=1364330140&sr=1-1&keywords=oreilly+sparql

You can read my paper on comparative study on Cloud Database...In this paper I have made comparision 16 Cloud Databases

Mahout does not contain code for that. Do you really need the inverse of the matrix or do you just want to solve a linear system? If you aim for the latter, there are faster ways then computing an inverse and Mahout also has tooling for that.

This question was originally asked by David Wolpert. In his worked he derived the No Free Lunch Theorem. This states that if you have no prior assumption on the nature of the problem we cannot expect an algorithm to be superior to the other. To my knowledge this theorem has not been proven incorrect. A good reference to introduce you to this theorem is the book by Duda, Hart and Stork "Pattern Classification" and the work by Wolpert. Hope this helps