File Systems - Science topic

You could use this: https://github.com/LlucSF/Raman2imzML

(R package to convert raman data in .txt format exported from WiRe 5.2(Renishaw) and FIVE 5.1(WiTec) into the imzML data format).

Hi Wisam

HDFS is a distributed file system specially designed to support big data. A block is a piece of data that can be distributed and replicated across multiple datanodes for fault tolerance and load balancing

Best regards,

RHP

There's a paper on how to build a timeseries database for edge computing, it might also be interesting to look at a few of the papers cited there:

better solution

$ ls /usr/local/cuda/lib64/libcusolver.*

$ sudo ln -s /usr/local/cuda/lib64/libcusolver.so.11 /usr/local/cuda/lib64/libcusolver.so.10

enjoy it

cheap network with less bandwidth may be used to transfer files

Norman Geist , thanks. I did not know the problem of DFT before. I will have a try :)

Unix and Solaris are the same thing. Here's how the kernel works, which will tell you how processes are prioritised.

IPFS and blockchains may work together well because of the similarity in their structure. In reality, Juan Benet, the founder of IPFS calls this a “ great marriage. ”IPFS is one of a few projects that are part of a community called Protocol Labs, which was also created by Benet. Some IPFS-related projects from Protocol Labs are IPLD (Inter-Planetary Linked Data), and Filecoin. IPLD is a paradigm of distributed data systems such as blockchains. This model allows for easy storage and access to data from blockchain through IPFS. Users willing to store IPFS data will receive Filecoin in return. IPLD allows users to seamlessly interact with multiple blockchains and has been integrated with Ethereum and Bitcoin. In short, IPFS connects all these different blockchains in a way that’s similar to how the web connects all these websites together. The same way that you can drop a link on one page that links to another page, you can drop a link in ethereum [for example] that links to zcash and IPFS can resolve all of that. — Juan Benet

I think yes

Both are same : store data indifferent location, also maintain replication for fail safe data, they have index for quick access, etc;

They differ by the way you can access or querying; they differ by data format u receive;

The fact is : database has to be on top of a file system;

Hi Goran,

Your best option is creating a windows service (.net) and using FileSystemWatcher class for file monitoring. It can be used to monitor network shares also. Anyway, as Rivalino said it can also be done creating file minifilter driver which resides in kernel space or try to use API hooking using Microsoft Detours package.

If its a straight binary dump of a data structure, the first difference you may encounter is endianess. It is straightforward to flip endianess. Unfortunately there are still some postprocessing applications (e.g. visualisation) that need big endian even on same system.

Dear Paramjit Kourç,

A distributed database is a database that consists of two or more files located in different sites either on the same network or on entirely different networks. ... A centralized distributed database management system (DDBMS) integrates data logically so it can be managed as if it were all stored in the same location.

Distributed File System (DFS) is a set of client and server services that allow an organization using Microsoft Windows servers to organize manydistributed SMB file shares into a distributed file system.

Regards,

Shafagat

The eigenvalues seem fine. There are a lot of them because this is a large system (1086 electrons), so I'll attach the corresponding OUTCAR file to this comment. The eigenvalues can be found starting at line 3557 for spin channel 1 and at line 18019 for spin channel 2.

We currently use only ARTAX Spectra, which I believe is Bruker software. The output format is PDZ23, or a CSV file with very little data about the actual device listed in the output. The output CSV files are two-block files, with the top block being basic two-column header info. The lower block is per-channel total photon counts.

I'd like a better format, with all pertinent header data listed, and in a machine and human readable form. I really expected there to be a standard format, like we have with state and federal water well and climate data, but I don't see anything like that on the internet.

The way Gaussian handles coordinates internally by default has issues when three consecutive atoms are arranged linearly. And your structure contains a lot of such groups.

There are a few ways around this: the easiest is adding the "cartesian" keyword – as mentioned –, which should be fine for most cases, unless you want to fix coordinates or run partial optimisations.

Other ways are adding "dummy" atoms perpendicular to the linear groups, or just reordering the Z-matrix to avoid having three linear atoms in a row in the definition of the geometry.

Justin Lemkul Thank you so much for your response. It was the hardware which made the difference. Its running fine now. Though, thanks once again for your insight.

"Smart cities, plants, and machines rely on smart analytics to put big data into context in real time and optimize decision-making. Powering today’s analytics tools are advanced algorithms, artificial intelligence (AI), machine learning, and digital twins that pull disorganized, disconnected, and “dark” (captured but unused) data into the mix of available information and make it actionable.

Strategic algorithms and cognitive computing improve operational acuity. Seeq software uses end-user-focused advanced computing algorithms to collect and analyze time-series data from manufacturing systems and process historians in real time. Trending, pattern and limit searches, outlier detection, modeling, and illustrative visualizations are among the capabilities that can be applied directly to the data sources without assistance from data scientists or IT experts..."

This is an important question.

A good start on an answer to this question is given in

More to the point, see

and

Hi Charlie,

Cheers. Will do!

Ben

very good links:

Hi Trinadh

You might want to think about textual analysis tools. You could start by running a simple word frequency count to get an idea of the frequency of words within the reports. You could use the linux command line to get you started. Simple tools such as head, tail, wc, grep, tr, sort, sed, cut, uniq and awk can get you a long way in a very short space of time. You could also write some simple Python scripts to carry out these tasks. Python will run on most O/S, is simple to programme and very fast in operation.

However, at the high end of the spectrum, you could use a tool like GATE (written by the University of Sheffield) to perform full textual analysis of any kind of text. You can apply your own list of keywords to the text, or you can use the tool to develop a keyword list based on the contents of the files. It provides you with a POS tagger and many analysis tools which can allow you to understand the sentience of the content of the file. This is a serious tool, and not recommended for amateurs, as there is a steep learning curve. You will also need a thorough understanding of Natural Language Processing to get the best out of it. I have added a link for the GATE software to let you see what it can do.

I hope that helps.

Regards

Bob

Dear Jyotima

Use this program... I think its the best choice ...

Hi Tripta,

I think you are talking about .hdf5 file. You can read it by using ENVI software.

I found a link-

Linux has an interesting relationship with file systems. Because Linux is open, it tends to be a key development platform both for next-generation file systems and for new, innovative file system ideas. Two interesting recent examples include the massively scalable Ceph and the continuous snapshotting file system nilfs2 (and of course, evolutions in workhorse file systems such as the fourth extended file system [ext4]). It's also an archaeological site for file systems of the past—DOS VFAT, Macintosh(HPFS), VMS ODS-2, and Plan-9's remote file system protocol

compression a file with out changing its bit rate is better.......

Can you please write something about PACELC or CAP here? 

There are also the so called extended file attributes,

like album, title, duration...

They seem to be added in various file systems.

What also could be done is to compute statistical data

and assign the files to clusters. I faintly remember

that it was possible to recognize componists by the

zipped version of their recorded music.

Regards,

Joachim

Cloud storage is the super set of cloud file. 

Example of Cloud Storage : S3, Database as Service, Elastic Block storage 

It seems the header file does not tell you much about how the data is stored in your .dat file. Did you check the Matlab file exchange to see if someone already wrote a converter/importer for your data?

Thank you Mahdi, glad to hear it was helpful.

As the additional points raised by Mark and Simon show, creating an efficient parallel system involves a number of subtle details, that was the main message I was trying to get across :) Therefore, you have to keep an open mind when reviewing your performance results. It is not always about cranking up the power.

Just a note, reading about your system and Simon and Mark's comments I wonder if the map-reduce paradigm would work for you. Hadoop is internally taking care of some of the details of juggling files in a distributed file system (http://hadoop.apache.org/). The low level considerations will still hold of course. Just an idea.

All the best

I met a manager who wanted to delete information older than 5 years from their database with the expectation that their information would be faster. The contractor that met the manager explained that it would involve de-linking references from old records so it would be cheaper to simply invest in newer hardware.

Some systems have an archival mechanism to remove old data; the cost of removing old data would therefore be dependent on whether the system allows the removing of old data or if it needs the software vendor to intervene. For example, one could easily sort an Excel sheet by date and remove old records but with an information system, there could be implications of deleting data from one or more tables without consulting the vendor.

.

reading this might help you :

.

Is your initial file a HEALPix FITS file and what do you mean with 'normal' FITS file? Do you want to use a different projection and have a single FITS image extension?

I'm sceptical about the existence of a general

multimedia database theory, because the number

of possibilities is so great. A historical paper

that forebode the internet and multimedia

is "As we may think" by Vannevar Bush.

A strategy that seems reasonable to adopt

in multimedia information systems is to

store the data in their raw, original format

(not in SQL blob fields) somewhere as

files and to store the meta information

in XML (filenames, urls, filetypes, length,

checksum...).

In order to ensure fast access to the

data stored in an XML node, the node's

content could be indexed, e.g. with

a suffix tree. Erasing a node could be

implemented by overwriting the content

in the XML file with blanks. Updating an

XML node would involve deleting the

old content by overwriting and to

append the new content at the end of

the file. The disadvantage is that

indexing is an offline operation wheras

in RDBMS records are indexed immediately.

Regards,

Joachim

Marcos - thanks - can you suggest a specific SRM I should look at?  A paper or open source?

Dear Huda,

I attached two Matlab scripts to import and show a .stl file.

Best regards,

Luca

@Hossein: numpy.getfromtxt won't work. The input data is not formatted correctly for this type of read. You could, however, reformat the text file to allow for simpler reads, e.g.:

f = open('Image Data.txt')

d = f.read()

f.close()

import re

d = re.sub(' ', '\n', d) # replace white space character with newline

f = open('Image Data Formatted.txt', 'w')

f.write(d)

f.close()

Now you can easily read the tuples into Python with numpy:

my_data = numpy.loadtxt(

fname='Image Data Formatted.txt',

delimiter = ','

)

shape(my_data) # (1800,3)

I would suggest implementing it using a Filesystem in User Space (FUSE) under Linux (see http://fuse.sourceforge.net/). This way you can use normal development tools, debuggers etc. on your code (and it won't lock up your system if you make a mistake). You may also code in languages other than C if you wish. The API is well-documented and there are many example filesystems implemented in this way. You can benchmark performance of your proposed filesystem using well-known tools such as bonnie++, or using real filesystem-sensitive applications such as big databases, squid http proxy etc.

Alternatively you could look at some simulated operating system; maybe something like MOSS (http://www.ontko.com/moss/) might be suitable.