Cluster Computing - Science topic

For those still interested, a new desktop version of GaussMem is available for calculating the amount of memory required by Gaussian calculations as a function of the type of calculation and the number of processors. It can be freely downloaded at the program page:

Quiero monitorear múltiples gráficos de convergencia ya que múltiples problemas se resuelven simultáneamente. Pero solo puedo monitorear un solo gráfico de convergencia... Por favor, dame un consejo.@

Berkay-Sungur: I DID NOT say that a Beowulf Cluster would be unsuitable. What I did say is that one built from small computers such as RaspberryPi or BeagleBone would not be suitable.

What you describe would require what I did suggest: A Beowulf Cluster built from reasonable personal computers, a switch, and cables that nobody wants anymore. I've built several of those. The computers do not have to be homogeneous, while I do recommend sticking with all 32-bit or all 64-bit. Whether a computer is a desktop, floortop, or laptop would not matter, as long as they have at least one ethernet port. It is possible to mix machines with different word sizes but that requires additional work to be done.

I've built several such clusters. The first one used machines that were pulled from a scrap pile. Some were put together from parts coming from that same pile. That bunch was all 32-bits of varying CPU speeds. One node was even less than 1mhz. Another was so old it eventually caught fire but was easily replaced. The software would allocate workload to a node according to it's cores processing speed. Node-level software allocated threads according to the number of cores. Thus, the advantages of distributed and shared processing and memory were brought to bear.

Another cluster was built from scrap to mimic the one used in a customer facility that could not be remotely accessed. We rented time on our cluster to a company that needed a remote testbed whose results could be directly installed manually on the customer's facility.

Beowulf clusters can be very handy as a laboratory's computational needs grow beyond what one computer can reasonably handle. All of the clusters I have built required NO capital outlay.

Dew computing is very trendy nowadays. You can check this very recent paper on this.

FYI: I deposited the R script code on Zenodo. I hope that it would be of any interest to you!

Shahid Ali If you were able to install it, please help me out.

It's not running on other CPUs.

I think, their are some mistake in the comand.

Thank you @Masab Ahmad

If you want to do research then just try this: https://github.com/lalithsuresh/absim/tree/table and paper: "C3: Cutting Tail Latency in Cloud Data Stores via Adaptive Replica Selection"

The book

" Experimental Design" by Federer

will be helpful in this situation.

I think Vahid already posted the answer referencing stackoverflow [3].

It is suggested to use a certain implementation such as mt19937.

Apparently R has a package which interfaces the dieharder suite (http://webhome.phy.duke.edu/~rgb/General/dieharder.php):

Use a stack data structure. With operations

Push(...), Pop(...), IsEmpty().

Many programming frameworks come along

with ready-made stack classes.

Regards,

Joachim

As I understand, Hadoop replicate data based on rack awareness.

you can read more about this here:

Thank you so much Vladimir for your information and the useful link. No, it is not really important how many clusters I have, I just want to find the best clusters. Thank you so much for the useful link as well!

Best regards,

Manijeh

If you are trying to do a virtual screening, instead of assigning maximum number of CPUs per job, running maximum number of parallel processes on the cluster might be a better option. There are several ways to achieve that as discussed in the attached thread.

I'd highly recommend using a higher timestep for your goal of getting the most ns/day possible. I don't recall the details but a colleage of mine is using a timestep of 6 fs, which is a 300% speedup compared to your setting of 2fs.

Have a look at "virtual sites" or "hydrogen mass repartitioning" in  context of gromacs

Some remarks:

1. Use the Krylov iteration method (Lanczos) for approximately solving the auxiliary linear systems.

2. If your matrix is diagonally dominant, use the Jacobi preconditioner. If not, try use the Gauss or Gauss-Seidel preconditioner. Then you do not need the matrix factorization (as in the case of an incomplete LU preconditioner).

3. Use the deflation of a computed eigenvalue to compute the next one. I suppose here you are not interested in eigenvectors.

If you're interested in pure QM, there's really no need to make clusters (unless for the reason of using a queueing system, which helps in managment of jobs or for larger storage). The reason is that QM jobs usually scale pretty bad with the number of processors, and it rarely makes sense to run any jobs with more than 16 cores. Even if I use a cluster for QM calculations I usually limit my self to 8 or 16 cores and make sure that they are on the same node, to make the speed optimal.

There is currently a number of very fast and quite reasonably priced (well...) processors with 6/8/10/12 cores and you can build very officient double-processor machines using them, which should handle most of QM calculations.

Hi,

I'm not sure to fully understand your question, but it wonder if you are not mixing several concepts.

Indeed, unless a an application runs with root privileges, it cannot open or close a port itself. Ports can be opened or closed at the OS level, for instance using its firewall UI. In other words, MySQL (or any other app) doesn't need to run for its related ports being open or close. In fact, MySQL will not be able to run properly if the port(s) it uses are not open before it is launched. Thus, checking if ports are open or close will probably not be a solution for you. If I'm right (but again, I'm not sure to have fully understood your question), you need to check the traffic on the (open) ports. A right tool to do this is tcpdump. You'll find several examples around, for instance at http://superuser.com/questions/604998/monitor-tcp-traffic-on-specific-port

Hope this helps.

Thank you all. Happy new year!

Hi,

Apart from above mentioned links, I would like to remind one thing, please dig the literature for bench-marking of the similar system as NMR being sensitive property to calculate.     

According to your description your problem originated from WMware network's configurations, not from Hadoop itself. In case you are using two machines you need to use the same network card with enabled DHCP on it, but on machines themselves instead of using dynamic IP address you need to configure static ones in the same IP range. After you will get both machines "pining" each other via static IP adresses, you will need to configure SSH (share common key between machines), then just use Apache Ambari to install your desired Apache Hadoop cluster. 

Good luck!

thanx jose...

I am having the same problem, I am working on DS 2016 on windows 7, been trying to dock using flexible docking but cpu utilization was 20% , re-ran the Job using parallel processing parameter> Server:local host> process 8 , and now its 50% utilization across all 16 threads I have

Bascially, you should know the oxidation state of copper in your system. 

Then, check a number of electrons what you have interestted in. ( odd ? or even?)

Then you can calculate multiplicity with consdering a charge state.

number of clusters in each direction=16,cluster radius=300m, number of vehicles in cluster=20,random distance between vehicles..

This is an industrial solution to your query. it is the first way to start  RNA analysis;

take a look to these  attached files.

Good luck,

Consider two components in standby configuration. Component 1 is the active component with a Weibull failure distribution (beta=1.5, eta=1000). Component 2 is the standby component. When Component 2 is operating, it has a Weibull failure distribution with beta=1.5 and eta=1000. Note: Even though Components 1 and 2 have the same distribution and parameters, it is possible that the two can be different. For the quiescent distribution, consider three different scenarios:

1. Same distribution as when in operation (hot standby).

2. beta=1.5, eta=2000 (warm standby).

3. Cannot fail in quiescent mode (cold standby).

What is the system reliability at 1000 hours? Note: For this example, we will only consider the non-repairable case, i.e. when a component fails, it is not repaired/replaced.

The reliability of the system at some time, t, can be calculated using the following equation:

R1 is the reliability of the active component

f1 is the pdf of the active component

R2,sb is the reliability of the standby component when in standby mode (quiescent reliability)

R2,A is the reliability of the standby component when in active mode

te is the equivalent operating time for the standby unit if it had been operating at an active mode, such that:

Solving Eqn. (2) with respect to te, you can obtain an expression for the equivalent time, which can then be substituted into Eqn. (1).

Dear Ariadne,

thank you for the link to the book. It is very interesting but in the frame of building computer cluster it prefers standard solution with LGM.

Best wishes

Radoslaw Jedynak

Before I start I want to make it clear I work for Intel and was the original lead architect of the Intel Cluster Ready specification.

Specifying the cluster is only part of the problem.  It needs to designed, an instance of the design created and the health of the same needs to be taken into account.  All of these phases make your total experience good or bad.   Since, there was many issues that came to bear int he cluster space, Intel wanted to make it easier to the end users to succeed.  Thus Intel developed a cluster architecture spanning small to enterprise and even Top 10 style system.   This architecture, Intel Cluster Ready (aka ICR), is available to anyone to download (see:   http://www.intel.com/go/clusters).  The idea is to solve two problems:

1.) Make is easy to purchase a cluster that is known to work and keep working.  -- i.e. you vendor creates a system that has been correctly integrated/ known to work when arrives at your site and will continue work well in the future.

2.) Ensure that the application codes that you want run can be installed and will function on that cluster.  [ISV's that are part of the program, validated that their codes "just work" and do not violate architectural considerations -- i.e. installing application A does not some how reek havoc on a previous installed application B.

But an architecture only goes so far, Intel has created a tool, called the Intel Cluster Checker which makes available for free to its OEM & manufacturing partners to help them in three ways:

1.) Certify that design will actually meet specification (design mode)

2.) Verify in manufacturing that the cluster actually is an instance of that design (manufacturing mode)

3.) Validate/Diagnose (health mode) that the current running cluster has not deviated from they way it was when it was installed (entropy has not take over). [The partners are required to deliver the tool to you the user, so you have it keep the cluster running properly]

I highly encourage you to go the web site and get the documents that are there.   Then work with your specific vendor and make sure they put together an ICR compliant cluster (if you want, do what may users do and add ICR compliance and certification as a requirement to your bid).  If they do not already have it, your vendor can get the cluster checker tool from Intel and use it to create a system for you.  Note they are required to deliver a copy of Intel CLCK & it's associated "fingerprint files" on the system as part of certification, so you have in the future to keep you system in top health.

Best wishes,

Clem Cole

HPC Architect, Intel

A note about Genuine Intel vs "or compatible"  and the difference between and architecture and instance of that architecture.    As the manufacturer of the CPU chips and creator of the specification and associated certification program, ICR defines the use of Intel*64 based processors in ICR certified clusters.   We can not speak for the validity of using non-Intel manufactured CPUs in the cluster.   Thus the specification calls for Intel*64 (aka Xeon or Xeon Phi) processors.   When the Cluster Checker tool runs, it will detect and warn of issues where the cluster instance deviates from the specification, from incorrectly set up IB connections, missing software components and the like.   If the tool detects a processor that is not what the specification defined a warning will be issues.    That said, ICR is an >>Architecture<< and you are obviously free to pick whatever components make you comfortable/you see as best for your project/meet your needs.   However Intel's certification program, by definition, can not certify an instance will function that deviates from the specification, which again,  includes a cluster made with CPUs that we do not manufacturer. 

interesting!

well, the problem is that HPC machines are normally designed to not throttle - if they did throttle, the only interpretation is that the heatsink is inadequate, or the airflow, or the incoming air temperature.  further, current HPC chips (say, e5-2690 v3) have a kind of built-in throttling, in that you can only achieve max clock by using a subset of cores.  to realy take advantage of improved cooling, you'd have to somehow get Intel to let you exceed this programmed TDP-based capping.  and as an HPC person, I'm not sure I'd go for a product like that, since although you might support higher clocks than normal, any chip's power-performance efficiency drops off as you really push it.  (a pitch based on reducing cooling costs would be more successful, I think.  and/or improved cluster density...)

I also note that the range for Intel's "turbo" modulation is about 10%.  that's not a knock-your-socks-off kind of competitive advantage...

A grid is a collection of resources. Those resources might be clusters or clusters plus other resources. In general, a grid is a geographically dispersed and more importantly organizationally and administratively diverse collection. I.E. a virtual organization. It is usually about more than just scheduling jobs on compute clusters, although for some grids (such as NSF's TeraGrid) that turns out to be the most important aspect. It can be a mushy idea since it was taken up and used as a marketing term, often by folks you didn't fully grok it. The classic paper as a reference is http://toolkit.globus.org/alliance/publications/papers/anatomy.pdf

Hi!

Regarding your question, I would suggest a density based method.

Normally, in density based methods, you do not set a number of clusters, but you configure a threshold density and a minimal number of neighbors.

DBScan works fine to detect clusters with similar density. However, if you want to detect clusters based on the local/neighboring density I suggest Optics or any further improved algorithmn of the Optics family..

This image shows clusters with different density:

There are many implementations of Optics. For Java, you can use Elki oder Weka, for example.

If you want to speed up the process, you can use a index structure for your neighborhood queries, e.g., quadtree, r-tree, kd-tree.

Best regards,

Robert

In order to run the make_ndx command in batch I have written a very useful script in Perl. You can find it at my repo https://github.com/dcorrada/RICEDDARIO

Just a different perspective on Stefan's answer: hypergraphs would be projections of the incidence relation of the concept graph, so you would be losing structure by using them. 

Incidentally, if you haven't seen it already, the Applications book that Stefan linked has a paper dealing directly with Conceptual Graphs and FCA.

I can tell you that by setting hop count limit of broadcast messege.

Dear Joaquim,

As you know CCSD(T) is too time demanding. I think it would be better first you submit a CCSD calculation and find how many hours one step of CCSD takes time,when you have it, multiple the time by number of occupied orbitals, so you have a time estimation on how many hours CCSD(T) takes time with respect to mem and nproc you have specified. For correlated calculations you need to increase RAM and cpu both. If you are doing your calculation on a cluster,specify more cpu to have more memory, and I suggest you that it would be better if all the cpus are on same node.

Ahmed Farid has done a good job of identifying the main decisions you need to make. What you decide affects the answer to your question.  ARM processors generally consume less electricity and generate less heat than x86 processors, so which you choose will affect your need for power and cooling.  

One other power-related issue is how reliable your local power grid is.  If it is not very reliable (e.g., power failures or brownouts), you'll want to include a battery-based uninterruptible power supply (UPS) into your design and use it to power your cluster, or you'll lose all your work in progress any time there is a momentary power failure.

If you want to build it from inexpensive system-on-a-board components (e.g., Raspberry Pi), I would look at Nvidia's Jetson board.  It's more expensive than the Pi, but has a quad-core CPU and a 192 CUDA core GPU, so it delivers quite a bit more processing power. There are other options out there too.

The networking decision is an important one that depends on the nature of the programs you'll be running.  If the programs running on your system will be doing lots of communication between the distributed processes, you want a high-speed low-latency network, or those processes will be doing a lot of waiting for messages to arrive. 

One other question you need to resolve: do you need 200 processors (CPUs) or 200 cores?  With 10 or 12 core Xeon CPUs, you can build a 200 core system in a relatively small form factor (e.g., half a rack).  The challenge becomes providing enough memory and network bandwidth to keep them all busy.

And once you have  designed your cluster, built it, and installed it in a room with sufficient power and cooling, it is very helpful to have a system administrator to maintain it.  For a small cluster, this can consume 1/4 to 1/2 of a full time equivalent (FTE) position, depending on how many people are using it, requiring software installation, etc.

Dear Sai,

Maybe Silhouette plots can help you:

Rousseeuw, Peter J. "Silhouettes: a graphical aid to the interpretation and validation of cluster analysis." Journal of computational and applied mathematics 20 (1987): 53-65.

Kind regards,

Olivier PARISOT

In addition to Simon's response, if your problem can be decomposed in small pieces, you can use the map-reduce approach. For computing in many computers, you also must distribute the jobs, for example, using http://discoproject.org/

Hi Paul,

you need to find saved project file of type *.spj. When you open it - all your result will be displayed and available in Parameter Set.

Yes, I agree with Jian.  In  ADF, you may be able set the oxidation states manually for specific atoms.

Thank you for help. I could find the error in ",dat" file.

I believe the only accurate method would be alignment-based phylogenetic analysis based on likelihood (e.g. RAxML on a public server such as CIPRES for such a large dataset), combined with a GMYC approach. Anything else, including and particular clustering and NJ methods, have been shown to be inaccurate, sometimes substantially so. Also, if you actually want to TEST barcoding accuracy, you need a testable expectation. For example, you have a set of barcoding sequences for KNOWN species (or at least species-level clades), based on a reconstructed phylogeny. Then you randomly select a sequence and blast it against the entire dataset and see how often the correct species is blasted. That is kind of trivial but that would be the only way of testing. If you have a set of unidentified barcoding sequences and just want to see how many distinct species you have in there, that would not be a barcoding test but an assessment of species delimitation. A very quick approach to sorting out species would also be using MAFFT with the sorting option, which will align the sequences and also sort them based on similarity. It is not 100% accurate but usually, especially if all sequences are complete, does a great job in outlining possible species-level clades even before running a tree. And 4000 sequences can be easily done in MAFFT.

David - your assumption is incorrect. what a Virtual Machine allows you to do is run multiple "computers" simultaneously on the same hardware. From your description, it sounds like you could *replace* all the other computers around the house by running VMs on your main 24 core server. You could easily allocate 3 cores for one Virtual Machine, 1 core for another, and so on - the only limitation would be memory and the performance you are willing to tolerate either on the main system or inside the Virtual Machines.

Any other kind of sharing amongst disparate machines would be done via network protocols (CIFS or NFS for filespace). There is a commercial product, ScaleMP, that supposedly allows you to aggregate memory from several different machines into one larger configuration, but again - that is not what you are describing.

I did a comparison of cloud HPC approaches including Amazon ECC here -

http://www.ibm.com/developerworks/cloud/library/cl-ind-cloud-e-learning2/ and

I think you can build a modest cluster of Linux machines yourself and get more out of it if you plan to use for a good length of time (say 2+ years). Some options to consider include GPU co-processing and use of CUDA and/or OpenCL as well as distributed computing tools (noted in the paper). CentOS is a great choice, but for ease of installation and use I like Ubuntu 12.04 LTS myself - it's just easier to install and configure with tools like CUDA, OpenCL, etc.

I suspect your workload would benefit from a vector co-processor and CUDA as well as GPUs for rendering. However, if you just need lots of cores, another option that may be of interest is MICA (the Intel Xeon Phi co-processor).

The advantage of AWS EC2 and other pay as you go HPC options like Penguincomputing (http://www.penguincomputing.com/services/hpc-cloud ) is that you can solve a problem and not own the equipment, but again, if you in this for the long haul, that's perhaps also the disadvantage - the idea of HPC on-demand is an interesting compromise you might consider - build your own small cluster and then use Cloud HPC for really big runs - this is my favorite approach. I find that I need to develop and test on my own small installation anyway (or it goes best if I do) and then use university resources or Cloud HPC once I'm ready for production runs.

As far as storage, Linux software RAID is an option along with Ceph if you're willing to pull together a JBOD and do the SAN/NAS configuration yourself.

If you build your own small scale system, I find it helps you make better use of Cloud HPC since you'll know better what to ask for and look for from any HPC on-demand service. You might find your private system is good enough.

Hope that helps.

Hi Sudhakar,

Have a look at this paper:

Chen, W.-Y.; Song, Y.; Bai, H.; Lin, C.-J. & Chang, E. Y.

Parallel Spectral Clustering in Distributed Systems. IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, 568-586

MPI is a message passing library interface specification for parallel programming.

MapReduce is a Google parallel computing framework. It is based on user-specified

map and reduce functions

It also says: "In general, MapReduce is suitable for non-iterative algorithms where nodes require little data exchange to proceed (non-iterative and independent); MPI is appropriate for iterative algorithms where nodes require data exchange to

proceed (iterative and dependent).

Best Wishes,

Ashkan

Rathers waited for 2 hrs to get result but no outcome

I received an answer to the topic on SeqAnswers. I'll paste it here just in case someone is interested.

Devon Ryan:

"Since tophat isn't written to use MPI, the instances run on each node will be blind to each other. There is no way around this without rewriting the program (actually, you'd need to rewrite bowtie as well). If you really want to use tophat, then just split your fastq files into small enough versions and then push a large number of tophat jobs onto the cluster (normally one would create a script to do this). If you're going that route anyway, just switch to STAR (you might have enough RAM, I'm not sure) and you'll get your results in a fraction of the time. The last alternative here would by Myrna, which seems to be designed with this sort of scenario in mind.

You'll have the same problems with cufflinks, namely that increasing the number of instances with MPI won't decrease runtime since the instances won't talk to each other. Your best bet there would be eXpress or to just use count-based methods. "

.

minhash is indeed a locality sensitive hashing scheme when locality is defined according to the jaccard similarity

see (by increasing completeness) :

http://en.wikipedia.org/wiki/MinHash#Jaccard_similarity_and_minimum_hash_values

http://www.cs.columbia.edu/~coms699812/lecture1.pdf

http://infolab.stanford.edu/~ullman/mmds/ch3.pdf

and

http://www.ucd.ie/t4cms/CeADAR_RS_Boydell.pdf

for an example of implementation

.

Sudhakar,

you will find quite a lot of litterature out there if you search for the terms "Beowulf" and "MPI". I am supposing here that you are looking for number-crunching (High-Performance) clusters, and not High Availability - which would be another totally different use case.

Some "classical" places to start out are the web page at ClusterMonkey, http://clustermonkey.net/ , and the mailing list archives at Beowulf.org, http://beowulf.org/ .

As for the practical aspects of system engineering, I would suggest taking a look at the Centos and RedHat administrator documents, http://www.centos.org/docs/5/ and https://access.redhat.com/site/documentation/Red_Hat_Enterprise_Linux/ . Quite a lot of this stuff is relevant to your query, both for High-Performance and for High-Availability.

Cheers,

-Alan

I suggest two things. First, perform a formal test to see if your data has evidence that clusters actually exist. Second, use bootstrap-aggregation to evaluate the robustness of your results.

Test for existence: http://www.ncbi.nlm.nih.gov/pubmed/12424117

Bootstrap-aggregation: http://www.ncbi.nlm.nih.gov/pubmed/12801869

What do you mean? The behavior of any batch system is knowable exactly as much as you know the distribution and properties of running and queued jobs. Do you know the actual runlength of the jobs? Do the jobs have differing resource requirements (cores-per-process, memory-per-process, processes-per-node, etc)?

Depends what you are trying to accomplish. For simply large scale data processing tasks take a look at Hadoop. But if you are doing clustered parallel processing in more time sensitive applications then Storm Cluster is great. Both can run out-of-the-box on AWS. If its simply a matter of load balancing, you can do this natively on AWS to manage load across VMs connected to an Elastic Load Balancer.

OpenStack is one of the best available option for now....as it open source and freeware we can do all the type of R&D upon it. Its implementation is little bit hard but we can try to understand it .. here is the refrence link ...http://www.openstack.org/

The function meandist finds the mean within and between block dissimilarities.

With the function meandist(), there is the function mrpp (in vegan library) : Multiple Response Permutation Procedure provides a test of whether there is a significant difference between two or more groups of sampling units.

It depends on the purpose. Parallelisation over ethernet is not very useful, communication slows down the calculation. We use MPI on multicore computers so that ethernet communication is avoided. We just use computers with shared /home via NFS. On one cluster Grid Engine was used. It serves our purpose but may not be sufficient for CFD.

People have implemented Chi-sq before for comparison (i randomly googled it and found a couple of papers) but you can also try agglomerative and see the trends in the clusters instead of increasing k by 1. Then you can see at which stage the results match up. Hope that helps!