Data Modeling - Science topic

We collected data from six communities in one city with the help of the local waste collectors who weighed the trash before it was thrown into the garbage truck. It meant more work for the waste collectors so we made sure there was an incentive for them to do this.

Hi Dr Behzad Soleimani Neysiani . I agree with Dr Qamar Ul Islam .

Two functions fitted simultaneously to one dataset? Do I understand correctly? What is the objective function?

Pooia Lalbakhsh you may want to employ deep learning models such as LSTM and GRU.

Good luck

Dear Esraa,

I cannot fully answer your question since I am also struggling with this problem. However, what I do understand is that the metric used in the Chi squared test scales with the a scaling factor. This can quite easily be seen:

where a is the scaling factor.

A colleague of mine also mentioned that normally you use this test when you are considering "counts" and categories. I don't think this is the case with your example (?). Furthermore, this colleague also noticed that when you have more counts your uncertainty decreases, and thus you will see that your significance increases when the scaling factor increases. I hope this helps..

Dear Meriam Nefzaoui

This is a nice GitHub repository in R to do morphological analysis

You may also like to read this short article on morphological analysis

Some improvements can be made to Ian's solution especially if you are working with students learning programming and/or VBA.

That might be overkill for business analysts or people who just need something done in Excel, but an important step (I think!) for people wanting to learn better coding skills that may be applicable to other languages.

Suggestions welcome!

Yes, this too is a dynamic model and applying Arellano-Bond (or the related GMM-type estimators) would be perfectly adequate.

Pooia Lalbakhsh

I think that your post "The side effect of Data intoxication" and the question "how real data science can be understood, rescued, and correctly applied, particularly in industry ..." is a rare sober voice in a hyped world of buzzwords. The same question can be asked on artificial intelligence (AI), Big Data, machine learning, etc.

As for Data science (DS), I believe that its purpose is inferring some insights (quantitative and qualitative) from an empirical collection of various data obtained usually without preliminary planning (just because the data are available). However, data science does not offer a causal explanation of any relationships between the variables (factors, features) but it explores merely some empirical patterns, associations / correlation of the past data and attempts making a formal projection of these data to forecast some future events. (By the way, is the future forecast always possible?-See the attached presentation).

Now, there are two distinct areas concerned with using the data potential. One is

focused on Technology for storing, processing and managing large amounts of data of various nature- this is the current trend. This trend leads to fitting a company’s arsenal with data-savvy tools. Value is too often framed as something that increases solely by the collection of more data. This means investments in data-focused activities center around the tools. This leaves an organization with a big set of tools, and a small amount of knowledge on how to convert data into something useful.

Another area is Methodology for making business decisions using modeling and simulation based on data specifically collected to address some business problems. This is called Business Analytics (and more closely related to genuine Data Science). It is getting momentum but still beyond the radar for too many companies.

(i) Defining a business problem, (ii) identifying an analytic method (algorithm) or simulation approach, (iii) collecting data required to feed the algorithm, (iv) turning solution into the actionable managerial decisions.

In summary:

•Data should NOT be a starting point in business analytics • More data not always result in more accurate predictions • Analytics is NOT a side effect of collecting, keeping and presenting / querying data in general. • Data have business value only in a specific business context. •Currently there is a trend to focusing on data itself instead of focusing on business context and methodology.

You can take a look for more details at the (attached) presentation ' DATA ANALYTICS FOR SOLVING BUSINESS PROBLEMS: SHIFTING FOCUS FROM THE TECHNOLOGY DEPLOYMENT TO THE ANALYTICS METHODOLOGY "

Dear Sejad Ayyoubi , not explicitly, as far as I am aware of. However, it will report the estimates for the fitted model and therefore you will have the estimate for k (d%=100*(1-exp(-kt))). Check at "Dissolution data modeling" and then choose one of the first-order model options.

Kind regards, Luis

Many indicators are constructed as functions of several variables not necessarily strongly correlated. The ultimate validity condition is their subject domain meaning. Frequently, they are used to measure a certain distance to an ideal modelling condition. Correlational methods, such as factor or principal components analyses, are very good exploratory techniques but they are restricted to linear relationships between the variables. So, you have a lot of work to do on your own.

Check the power index in the denominator of your expression for the curvature. It should be 3/2 rather than 1/2. Good luck

Hi Dear;

Please check them up:

Good Luck.

, Why would you suggest it is not being advisable? any particular reasons or limitations?

Regards

Gaurav

One thing to keep in mind is that you can't use the original sampling weights anymore, since you excluded 370,000-726 observations from the original data set, so the the original sampling weights are no longer valid. Calculating new sampling weights for your sub-sample would be very hard to do. Also, when and why to use sampling weights, i found this very useful: https://blogs.worldbank.org/impactevaluations/tools-of-the-trade-when-to-use-those-sample-weights

You can sum the scores as you would do in a typical knowledge score, with 0 for bad/false or 1 for good/true answers, see example in:

I do not understand why you want to transfer them to a 0 - 100 score though, seems unneccessary to me.

How to use it? You can use it as a dependent variable and for example analyze predictors of software competence. If you have a learning program you can examine change or you can also do an RCT (one group gets the learning program) and compare groups

Best

John-Kåre

Hi,

I just scanned: Hirt, C, S.J. Claessens, T. Fecher, M. Kuhn, R. Pail, M. Rexer (2013), New ultra-high resolution picture of 1 Earth's gravity field, Geophysical Research Letters, Vol 40, doi: 10.1002/grl.50838.

It appears to me that GGMPlus is the most recent Free Air Gravity compilation utilizing satellite data. It should be convertable to Bouguer Gravity if you have a good elevation grid, and then convertable to Isostatic Gravity if you have a good isostatic correction grid.

Thank you very much everyone, very interesting comments and recommendations.

I have discussed with my supervisor and he recommended association rule but also to be validated with other techniques. I am now cleaning the data and trying to create categories for the factors.

Is there any credible source where i can find Meteorology terms guidelines if possible for Germany .

For example

The temperature range for Cold(chilly), sunshine, and Worm

Samer Sarsam I will do cross validation thank you .

Jesús S. Aguilar-Ruiz I would like to understand the relationship and also to predict . I will work the association rule but i will check also regression tree model. thank you.

Bojan Ploj I will discuss that with my supervisor

Milford Estay Thank you for your offer but for privacy reasons i am not allowed to do that plus i am new to data mining but i would like to learn.

Jorge Humberto Guanín Fajardo : Any particular supervised techniques in mind?

Thank you all for your help

Kib

What you need to do is keep all the conditions of the reaction constant and vary only one condition for different catalyst acting on the jatropha circus seed. For example you can keep temperature, pressure constant and vary concentration of different catalyst acting on different specified amount of the jatropha circus seed and the measure the amount of biodiesel produce per different catalyst concentration. The data obtain from catalyst concentration (serve as input); and that obtain from biodiesel produce per specified amount of catalyst (serve as output). Then the model will be obtain from input and output obtain in this regard. Thereafter, The simulation will follow. Thereby, proceeding to analysis and conclusion.

Thanks Wendel.

The applied regression in the article you sent is polynomial, wich doesn't work for nonlinear and nonpolynomial equation

Finally I find a solution for the fitting step, using the R nls function.

Primary I had to determine a weight for every point, based on variance and number of experimental repetition: W_i=N_i / v_i.

Finally I used the nls function, giving the weights input into the function:

start2 <- list(ym = 10000, b=0.01)            #my starting parameters values

n2 <- nls(Yield~ (ym)*(1-exp(-b*x)),         #my equation

                  start = start2,

                 data=mdado,

                 trace = FALSE)

There is an element of chance in the training process. In some software, you can get reproducible answers by using something like set.seed( ) in the R language. Using the seed number again with the same data will then give the same result. Then you can report the software you used with the seed. However in general the different outcomes will be close together, but as with sampling, you will occasionally get outliers (depending on the seed you choose). 

Python (scikit-learn + Pandas or Anacondas), and you can always call R from Python (https://sites.google.com/site/aslugsguidetopython/data-analysis/pandas/calling-r-from-python).

Pca has no limiting a priori hypothesis but you must

Be very careful in the choice of variables and statistical units

Relative role so if you have (reasonably) three times of

Blood glucose at 6 12 24 your variables must be g6 g12 g24 and

Your statistical units the different subjects if you have exercise

At lets say 11 15 40 hours who care you add other three variabkes

Named e11 e15 e40 each individual so will be a row vector with his

Correspobding values for each variable pay attention to fill all the

Variables and not to have voids in the matrix then go on (before

Obviously read carefully one of the many existing introductory

Articles or books to pca)

Hi, 

what is missing in your message is the subject of your thesis and its title, from which you can have a first idea about the context and motivation of your work. Is the department in need for a traditional data warehouse or are there any specific needs to be met? is the company dealing with structured data only? are there any 'data warehouse / big data" integration needs? what is specific with the company's data and needs that cannot be met by existing approaches of data warehousing?

I think that your starting point is rather the title and theme of your thesis. Once you have a clearer idea about what your company is looking for about data warehousing, you can draw the main keywords then drive your "state-of-the-art" and "state-of-practice" according to the keywords.

You are welcome.

They are first of grouped by key and that sorted. While processing key/value pairs in your reducer (data is passed into it in sequence fashion) you have to implement by your own 'splitting" of incoming data sequence. Here is example for you (it's in Python, but could be use with Hadoop Streaming) - https://gist.github.com/anonymous/f28f051bb2a7b1532d03

Hello,

Google just open sourced its deep learning library a few weeks ago. It's the same C++ library they use internally to train their different deep learning algorithms for image recognition, etc. Look at the following link for TensorFlow.

You could also have a look at DLib, also in C++.

Dear Botu,

I would like to add a note to the good comment provided by Peter.

There are different approaches/criteria to define an urban wayfinding model. However, I think that many times researchers forget that people do not have a God's view knowledge of the scenarios. Therefore, criteria such as shortest distance could nor be realistic if a person does not have a map of the scenario (i.e. city map) and skills to identify the shortest path in complex scenarios.

I have been studying this problem for building evacuation and I have submitted a paper on this topic but it still under review. Please contact me if you want a copy of this work.

My email is: lovreglio.ruggiero@gmail.com

Regards

Could you please state more precisely your understanding of "movement data" and "sequences"?

Dear Colleagues,

Good Day,

suppose you have the following data:

x=[0.9 1.5 3 4 6 8 9.5];

y=[0.9 1.5 2.5 5.1 4.5 4.9 6.3];

Curve fitting with polynomials is done in MATLAB with the polyfit

function, which uses the least squares method. The basic form of the polyfit

function is:

For the same set of m points, the polyfit function can be used to fit polynomials

of any order up to m-1 . If n = 1 the polynomial is a straight line, if n = 2

the polynomial is a parabola, and so on. The polynomial passes through all the

points if n=m-1  (the order of the polynomial is one less than the number of

points). It should be pointed out here that a polynomial that passes through all the

points, or polynomials with higher order, do not necessarily give a better fit overall.

High-order polynomials can deviate significantly between the data points.

 

p = polyfit(x,y,n)

x is a vector with the horizontal coordinates

of the data points (independent variable).

y is a vector with the vertical coordinates of

the data points (dependent variable).

n is the degree of the polynomial.

p is the vector of the coefficients

of the polynomial

that fits the data. 

p=polyfit(x,y,3)

xp=0.9:0.1:9.5;

yp=polyval(p,xp);

plot(x,y,'o',xp,yp)

xlabel('x'); ylabel('y')

See page page 267-274 , if you need any help just ask, please.

Hazim

How can I find the coefficient for fitting a curve in MATLAB? - ResearchGate. Available from: https://www.researchgate.net/post/How_can_I_find_the_coefficient_for_fitting_a_curve_in_MATLAB [accessed Jul 8, 2015].

Try and download the book from the attached link ( look for my answer there you see the pdf of the book)

sorry I do not have the expertise to answer this question.

Regards

Dear friend i have not used version 8 of EViews.  Pl pardon me.

Hi Ines, It is an interesting question. It would be even better if we had some examples or diagrams here to illustrate the issue. I teach (and do) systems modeling and many of of my students also do context diagrams, aside from data models, process models and object models. My general answer would be yes, it is sometimes possible to have the same concept in different models. If something is in the business context model but is  also a major entity that we want to record detailed data on, yes, it would make sense to have it in the data or database model as well. Location related data, for example, can be quite detailed. Best wishes and good luck with your work. 

Splitting your database seems like a very messy approach. Many database technologies support data access out of memory, and can also be very quick if your data is structured properly. I would suggest looking at HDF5 or something similar. Python has some good tools for this work with PyTables and Pandas. Good luck. http://stackoverflow.com/questions/14262433/large-data-work-flows-using-pandas 

Ok, answering my own question, using my answer in another place...

To convert the site from django-registration to django-allauth, I only needed to change "'registration'," to "'allauth'," in INSTALLED_APPS within settings.py (well, I also moved it closer to the end of the list), and all instances of id_username to id_login. All together, 5 files including a test and a css file. I used id_username as a name for an id attribute and changed that too, for consistency. 3 files (1 line per file) would be enough: settings.py and two templates.

I used just basic authentication, and the templates for login pages are in the project.

I also found that the blog (see a link in the question) worked for a short while during the last week. It has something more on installing and on copying the templates, which I had ready before I started the migration.

Tested in Django 1.6.

Upgrade to Django 1.7 is not finished, it needs more work...

Hi again, also see Microsoft Research's Automatic Graph Layout library:

In statistics, any model has its assumptions, it seems you use linear regression, and there are 5 assumptions about linear regression,

1.       All observation are independent

2.       The dependent variable Y has a linear relationship with independent factors x1, x2, …

3.       The variance of Y are homogeneous

4.       All x1, x2, … are constant without errors

5.       All x1, x2, … are independent

In other word, the residuals of the regression Y=a+b1x1+b2x2+… Yi-^YI  are independent and  identical normal distributed.

Hence you should check if is the distribution of residuals skew or not, rather than Yi

You should not use R^2=0.4 only to say if the model fit well or not. If your data size is large and a few independent variables are used, the R^2 could be low. If transformation is used, you need compare the R^2 between original data and transformed data to see how much are improved.

Low R^2 are due to missing significant factors, or other reasons.

Its depend on what do you need, no matter what the architecture, its go back to the purpose to build datawarehouse where we need to deliver database environment which can create best sql performance when access data from datawarehouse

you can create star schema or snowflake, top down or bottom up, its depend on how best performance your sql to access datawarehouse rather than from oltp.

To check the performance of your model you can carry out cross-validation by leaving one observation out and fitting the model using the n-1 observations and comparing the predicted value with the observation left out in some way using the new model and repeating on each observation in turn. (Many R software functions have facilities for carrying out cross-validation)

If the performance of your model is not satisfactory enough you could refine your model (by finding a more suitable model or removing outliers if exist for example) and repeat checking the performance of your model until you get a satisfactory model.

Find one of my Research work in which I performed Rota-rod model to check SKM Relaxant activity of Hibiscus rosa-sinensis. Hope it will give all the required details to you...

MISUSE OF LINEAR MODELING

Linear regression has often been misused to be the holly grail of proving relationship forecast. There is always a built-in tendency to conclude that Y changes at the rate of “b” for every unit change in X, reading from the linear equation: Y + a + bX + c. Actually, this equation is used for a rudimentary forecast given an array of X: (x1, x2, …,xn) and Y: (y1, y2, …, yn). Some financial or economic data may not be linear. For non-linear data, linear modeling may not be appropriate.

CORRELATION COEFFICIENT

When working with financial information, some data may not behave in a straight line fashion, especially cost functions or productivity. Instead of using linear equation as a starting point, one may look at correlation coefficient as a tool to measure the association between two data arrays. In so doing, use the following as a guide:

X quantitative: Y quantitative = Pearson correlation

X quantitative: Y ordinal = Biserial correlation

X quantitative: Y nominal = Point biserial correlation

X ordinal: Y quantitative = Biserial correlation

X ordinal: Y ordinal = Spearman rho or tetrachoric correlation

X ordinal: Y nominal = Rank biserial correlation

X nominal: Y quantitative = Point biserial correlation

X nominal: Y ordinal = Rank biserial correlation

X nominal: Y nominal = Phi, L, C & Lambda

The correlation coefficient interval runs between negative one (-1) and positive one (+1). The association may be read from this range. For forecast, if linear regression is not appropriate, then consider non-linear curve fitting.

NON-LINEAR CURVE FITTING

For non-linear data dispersion, it is necessary to look beyond linear modeling. One needs non-linear curve fitting. One approach to curve fitting is to increase the order of a polynomial equation. Basic definition of a polynomial follows: (1) the line equation y = a + bX + c is a polynomial of the first order. It is a best fit equation for two points; hence, the shortest distance between two points is a straight line; (2) the equation y = aX^2 + bX + c is a second order polynomial; it is the best fit of a curve that is bounded by three points; and (3) the equation y = aX^3 + bX^2 + cx + d is a polynomial of the third order; it is a best fit polynomial connecting four points. Each point referred hitherto is known as a constraint. Each constraint may be a point, an angle or a curvature.

ATTACHED FILE

A file is attached for exercise in nonlinear curve fitting.

i think you have mixed many things, autocorrelation is not a method which we can apply on a data, but it is actually data type. a particular variable may be autocorrelated or not. e.g. your current weight depend on the weight you had one month ago. actually when you take some measurements over a time of a particular variable (weight, height, income, any chemical process,or activity , speed of a vehical ) and it seems that current measurement have some relation with prevous one then measurement recorded are said to be autocorrelated. and if you feel successive measurement are independent, and donot follow any pattern, then there is no autocorrelation. Now non stationarity is related to time, measurements taken our time (whether autocorrelated or not ) have same distribution (paramters ,mean variance , covariance remain same ) over time then the variable is called stationary and opposite to it is non stationary. (stationarity has types but i tried to tell you simply just to understand). Now depending upon your variable you have to decide the autocorrelation, which is usually narrated in the literature that which variable is autocorrealated. and thn testing for stationarity and then go for further analysis. hope this will help