3D Reconstruction - Science topic

Not neccesarily,Other options like photogrammetry, LiDAR, or structured light scanners can do the job too. Each method has its strengths, like photogrammetry for detailed textures or LiDAR for super precise large-scale models. It all comes down to what your project needs and what tools you have.

Hi,

To calculate an average, first you need to make sure that the images are well-aligned and the subject/specimen has not moved between acquisitions, thus you may want to co-register your images (one option would be FSL flirt https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FLIRT/UserGuide).

For calculating the mean you could also make use of FSL functions, see e.g. https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Fslutils (most interesting for you are probably fslmaths and fslmeants)

BR

Peter

I will concur with Anton Vrdoljak

Dear Umesh Persad

Have a look:

1. https://github.com/modenaxe/awesome-biomechanics

2. https://github.com/shawnyuen/DeepLearningInMedicalImagingAndMedicalImageAnalysis

3. https://miccai2021.org/openaccess/paperlinks/2021/09/01/406-Paper2031.html

Kind Regards

Dear Navid Freidoonimehr

These links might be useful, have a lok:

1. https://github.com/ArsoVukicevic

2. https://github.com/ucuapps/CoronaryArteryStenosisScoreClassification

3. https://github.com/zgongkuang/IterativeCNN

4. https://github.com/syshin1014/VGN

Kind Regards

Raghu Ram Katreddi Thanks Raghu. Already very helpful.

Cheers.

Hi Liu Yang,

May I recommend using the CT scan processing software Slicer (https://www.slicer.org/). In this free program you can load in a series of tiff images of cross sections. The program can be used to convert it to voxels for all sorts of visualizations and analyses. Since these wouldn't be typical CT files, you would have to scale the z direction based on your histological section spacing. Also, I believe this only works with black and white photos so you may have to convert them to grayscale. You can also adjust the window of pixel values (normally x-ray attenuation factors) to select some desired range of densities (and in the case of histological sections - colors) and turn the 3D voxels into a 3D mesh (stl, obj, etc). There are lots of tutorials on youtube, as it has become popular for people to take their medical CT scans and 3D print them. Also, the input file format is really important. You will have to trick slicer into thinking your images are tomographic slices in tiff format. I suggest downloading a batch image processing program to convert and label each successive section in a folder.

There is also a similar program with a wide range of utilities called molcer (white rabbit co). The free version has some nice utilities, but you would probably have to pay for the full version to process histological sections as if they were tomographic images.

I hope this helps. Let me know if you have any additional questions.

best regards,

David

Muhammad Umar Thanks for the answer, but Mtex does not have 3D functionality, but for pure visualization it can indeed work.

If you are using Imaris to open these images (which you hashtagged in your question so I am assuming), then after opening the images, imaris gives you the option to click on the image using the cursor and reposition it the way you want it.

Uncheck the "frame" option to get rid of the frame. Take a snap and open it using imageJ, and in there you could re construct the xy axis (make sure you know the pixel size before you uncheck the 'frame' option, and draw the xy axis accordingly) hope this helps.

Waqas Tanveer Sorry if you select the most convineant software please share it to me convenient

thank you in advance

Hello Nima,

I am curious about your need, what type of images do you acquire: brightfield, fluorescence of some kind, other...?

I am currently working on an automated serial block face system which allows image acquisition and 3D reconstruction of unstained samples based on autofluorescence signal. This system could typically be useful for the question you ask. Maybe we can discuss?

You may look at datasets via https://www.visualdata.io/discovery and https://www.robots.ox.ac.uk/~vgg/data/ OR

Check the free Autodesk Meshmixer here: https://www.meshmixer.com. You can combine parts or separate them. Many useful tools included.

I use ITK SNAP

Maybe you can use a general purpose robotics simulator, like Gazebo, where you can create a scene and add two cameras and capture images. Then, add the speckle in a post-processing step

Not specially but similar approach could be applied

Hi, following a point-based registration approach seems to be the most viable option at the moment until machine learning is able to bridge the non-canonical views properly. Besides Francesco Nex's reply and linked research, there is an interesting approach by Zhu et al. (2020):

with the code and a respective test data set publicly available:

You are looking to close the loop. You need to find N registration parameters, and then apply loop closure constraint to refine the individual parameters. Kaess et al's iSAM paper provides a way to do a smoothing operation to reduce the error between the first and last frame. However, in your case you may be able to get away with a simpler iterative approach. Essentially you estimate registration in a series for scan 1-2, 2-3,...(n-1)-n. After that you swap the source and the template to find registration n-(n-2), ....2-1. You repeat this process a few times, every time applying the estimated transformation to the next point cloud. In a few iterations, the overall registration should get refined.

Hey Mariana Vargas-Caballero, good to hear from you and thanks for the pro tips! Regarding retraction speed, I too prefer a relatively fast retraction over a couple of seconds, starting slowly and gradually increasing pace if you see your cell is closing nicely on the membrane seal test. This has always worked fine for me; it’s just recently that I’ve been having the issue of sticky nuclei. A bit of positive pressure only helps occasionally sadly… When the lab reopens from the corona lock-down I’ll review everything and hope I find the culprit. If so, I’ll be sure to report back here. Be well.

You might also be better off loading your pyramidal TIFFs into a software more suited for what you want to do. ImageJ is freely available and is more suited for scientific analysis like you are trying to do, and I suspect it could open your files natively.

Hi Denis Korneev and Andreas Müller,

You can also try Dragonfly (https://www.theobjects.com/dragonfly/index.html) from Object Research Systems, it is free for non commercial use.

Dragonfly has many segmentation tools from active shape to deep neural networks. The following video, (https://www.youtube.com/watch?v=1WVlskyuw94) is a deep learning tutorial.

Do not hesitate to contact me if you want more information.

Mr. Reyes,

Thank you for your elaborate answer. I ended up with a solution where I converted each RGB pixel to HSV format and specified a Hue range to remove and ended up with an acceptable solution.

Thank you.

If your research is into the general 3D reconstruction then the following might be of interest to you:

# International Journal of Computer Vision

# IEEE Transactions on Image Processing

# Image and Vision Computing

Else, if the content is leaned more towards 3D reconstruction in the medical field then you could check:

# IEEE Transactions on Medical Imaging

Lambert Zijp,

While I completely agree with what you said, I was just pointing out the existence of cone-beam CT scanners. Thanks again for your respective reply.

Thanks a lot for sharing your answers. I think the scanning parameters are also important. How to use the proper voltage and adjust the source and detector distance all count for the final resolution of image.

Please read this book " Polygon Mesh Processing By Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre "

Hello Manu Francis ,

yes you can use STIR for 3D reconstruction check [1] and you will find STIR paper and open source program it will be useful in this topic.

there are a lot of 3d reconstruction algorithms working under STIR. so you have to google it "STIR 3d reconstruction"

Good luck

Mohamed Samir

references:

[1]

I'm not sure about the 3D reconstruction, but for histomorphometry you could use ImageJ with the plugin Map_BoneMicrostructure. It can give you accurate measurements of trabecular morphometry, and you can complement it with some of the basic package measurements such as integrated density.

Thank you Lambert Zijp

You can calculate the difference of distance between the original and the projected 3D point. This is called re-projection error.

Yes, you can convert a 3D model to a series of cross-sectional images using 3D Slicer. For example, you can load the STL file, then in Data module right-click to convert it to segmentation, then right-click on the created segmentation to convert it to labelmap. Post further questions on using 3D Slicer to discourse.slicer.org.

MeX is a good software. I believe there are some other programs to work with stereopairs. Buy the way, calculations and measurements are simple. If just a few points are of interest, it is possible to measure with no software at all (but then you have to read about the method).

Webplotdigitizer is the best and free. It works online as a well as offline

Structure-from-Motion is a technique that is most commonly used to created a 3D scene (point cloud) from a set of unordered images. There are many available open source software that can perform sfm for you. Examples are:

VisualSFM that can be found at: http://ccwu.me/vsfm/

OpenSfM that can be found at: https://github.com/mapillary/OpenSfM

@ Yuan Li: Please do check the below link

Moreover, I am sending you a ebook which may help you

Thanks

It helps if you have more pictures from different views. Here is a list that may help. https://all3dp.com/1/best-photogrammetry-software/

I recommend Agisoft... this is not free but if you are with a University you can get a discount. Hope this helps.

I don't know if he's active on ResearchGate but I know that @JonasAdler has done some work in this topic. His publication list should get you started and I know he normally has his code on GitHub too. Even if his applications don't exactly match your own then at least the code should still be good. Bear in mind that deep learning is dictated by having good training data, there is a nice low-dose CT dataset available online which is probably why a lot of people have gone for that first...

H Adrian,

Smoothness is lack of roughness and is related to the areal surface texture analysis in metrology. Follow:

Regards

Based on my experience with a multi pinhole SPECT, there is no conventional sinogram space as there is in a conventional SPECT.

In the multi pinhole SPECT that I used to work, there was 3 detector heads and the projections from each head was used for image reconstruction. Something similar to stiching method. However, these projections were called AKA sinogram.

3D Slicer is a free open source software that is a flexible, modular platform for image analysis and visualization, and useful for you.

Hi,

Maybe this question was already solved somewhere else , but I found a way quite easy that might be useful for someone.

1) Create a Surface object (if the threshold is quite high, you can include most of the information that is in the Volume object.

2) Select the object. Surpass --> Export selected objects --> Save as .vrml

3) Manually, change extension to .wrl

Now should be suitable for MeshLab and there you can save it as .stl.

Thanks Neils, I will!

Hi James,

There are several packages available with ImageJ that it sounds like you are aware of but there is a stand alone package called Vaa3D that I think will do all that you are wanting. I've used it for very simple rotational videos. It also allows you to load time stack information as well. It's available at the below link for free.

and a link to the original paper is here

Though it would just be an estimate, some work can be referred in the area of hand gesture recognition where 3D modeling of hand is attempted with a single reference (2.5D) frame. Despite that Its not general, might give some clue:

-- Kirac, F., Kara, Y.E. and Akarun, L., 2014. Hierarchically constrained 3D hand pose estimation using regression forests from single frame depth data. Pattern Recognition Letters, 50, pp.91-100.

-- Zhang, C., Yang, X. and Tian, Y., 2013, April. Histogram of 3D facets: A characteristic descriptor for hand gesture recognition. In Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on(pp. 1-8). IEEE.

Thanks JLSL for a quick insight. Indeed, I am using a cropped frame (128X128) for generating (x,y,z) where I can't see much depth filled really. The Kinect v1 originally produces 640X480 RGB and depth images and I tried generating (x,y,z) from full images but still the results are not very promising. Can you advice on what techniques can be employed to remove such effects? I did not understand how a computer can influence with its HW? Thanks

Hello, it appears (from the log file) that there are too few registered points in your images to begin with. I would suspect that either there is something inconsistent with the output of openMVG into the openMVS workflow or your set of images is not appropriately shot. Try another set/scene to see if the io works first.

Sorry for the delay.

Thank you all for your answers, I found them very helpful.

I tried stitching in one dimension and then stacked the resultant images with trakEM2.

Best regards

Hello Yingnan,

Just follow the tutorials in this order:

1. Alignment of 2D Physical Cross-Sections

2. Image segmentation

3. Surface reconstruction

an example is given in help/demos: "Geometry Reconstruction"

Analytical calculation+ PSF, Ref: Accurate image reconstruction based on standard-Gaussian-model fitted system matrix in multipinhole small animal SPECT imaging

Yes it can be. A bad reconstruction could be several reasons:

Tilt angle is very likely, higher tilt or second tilt axis would make a lot of difference.

If it is dense need to check you are getting beam all the way through at all angles. If it blocks the beam can get artefact in the reconstruction (like around gold fiducial markers)

if you are not using markers, try to. Alignment is much easier and more accurate. Also: Join the IMOD user group/list.

Suggest using OpenGL

1. set your camera so screen projection plane is parallel to the slice...

2. clear screen buffer as usual with glClearColor set to background color

3. Clear your depth buffer with glClearDepth set to Z-coordinate of the slice in camera space

4. set glDepthFunc(GL_EQUAL)

5. render mesh

glClearColor( 0.0,0.0,0.0,0.0 ); // <0.0,1.0> r,g,b,a

glClearDepth( 0.5 ); // <0.0,1.0> ... 0.0 = z_near, 1.0 = z_far

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

glDepthFunc(GL_EQUAL);

This will render only the slice for which fragments have Z==Slice coordinate. This can be also done by GLSL by throwing away all fragments with different Z. The DirectX should have something similar (I do not use it so I do not know for sure).

As most meshes are BR models (hollow) then you will obtain circumference of your slice so you most likely need to fill it afterwards to suite your needs...

What's the update ? @ Dr. Tatyana

Hi Lei,

I am not sure what is the main problem in your application, so will try to address each of the points you made in your query, hoping that what you need to know is inside these responses:

1) "The monocular slam's initial coordinate system is random and scale-unknown".

It is true that single camera does not suffice to obtain the scale of a scene geometrically reconstructed from correspondences in two (or more) views. In other words, the baseline (i.e., the vector connecting the two camera centers) has unknown scale/norm. So we usually pick a norm for this vector (usually 1 if no other information is given) and everything else fall;s into place. However, the initial reference frame is not random. It is simply taken as the identity orientation and the zero-vector [I | 0] in terms of position/translation, simply because there is no other reference to be expressed it in terms of. So this is easy. Coordinate frames have a meaning only in relation to other coordinate frames.

The usual SLAM initialization will entail the recovery of a homography (see PTAM) or essential matrix (see my technical report here: https://www.plymouth.ac.uk/uploads/production/document/path/8/8593/Relative_Pose_-_George_Terzakis.pdf ). Of course, the baseline will have arbitrary scale, which typically is chosen as 1 and the rest will fall into place.

2) "But I want to know how to transform the initial coordinate system to another coordinate system(from a marker, like chessboard)".

Suppose that you have two coordinate frames placed at positions b1 and b2, with respective orientations R1 and R2 (containing the direction unit vectors of each frame as columns), then a point M1 in frame #1 has coordinates M2 in frame #2, related with (using MATLAB syntax):

M2 =R1*R2' * (M1 - ( b2 -  b1 ) )

Now, if the first frame is taken as the global (or world) reference (hence, R1 = I3 and b1 = 0), then the transformation becomes: M2 = R2' * M2  * (M1 - b2).

In the case that you are describing, you have to use the general case, since the PnP will give a pose in terms of the local coordinate frame of your landmarks.

3) "But the mappoints reconstructed by the marker during the SLAM's initialization were untrusted.And the graphic optimization would change mappoints' position to make a smooth trajetory by minimal the projection error."

You are using the PnP to obtain camera pose (i.e. orientation and position) from known points. You mention that these points are "untrusted". I suppose you mean that their true position is uncertain (?). I suppose you mean that their *tracked projections* on the first two images in the sequence are uncertain. Yes, that is correct and moreover, it would propagate uncertainty into your camera pose estimate. This estimate can only be improved by tracking these points further in subsequent frames, repeat the PnP for these camera frames to obtain the respective camera poses and then run *bundle adjustment* to improve the camera position estimate (usually BA improves both camera pose and feature locations, but since you know your landmark locations, BA should improve only the poses and potentially the 3D locations of features that have been detected in new frames, but did not belong in your initial set of landmarks).

Feel free to ask more, or take a look at my PhD thesis (chapters 2-3) for more information.

I really hope the above is helpful.

 great, I see. Thank  you for the suggestion. 

Peter, I encourage you to try out Stratovan Checkpoint as well. I wrote the IDAV Landmark Editor software while I was a post doc at UC Davis some years ago. I left the University and started Stratovan and built Checkpoint is essentially the "next" version of Landmark. Checkpoint can do many of the things that Landmark does and much much more. It's more robust, can work with CT scans, can handle larger data, and is easier to use.

I segmented the coronary arteries using the region growing method as proposed by the article attached with this Question. I generated the 3D point cloud by using Isosurface command in MATLAB. Attached image shows a portion of point cloud, in this image,

1. The grey region represents a portion of segmented coronary arterial tree.

2. The black sports, represents centerline points of the segmented coronary tree.

3. The red points are the points for which the following equation holds

                          nx(x - xc) + ny(y - yc) + nz(z - zc) <= T

Thank you for your suggestion, I am going to read it.

Best regards,

Amal

if you work with matlab try to use the instruction plot3()

For non contact 3D measurement, LIDAR is the most accurate. Dense LIDAR is very expensive, sparse (such as automative) is cheaper, but will return only a sparse set of points with true depth, in the scene, Other less accurate methods are structured light, time of flight and stereo/multiview camera 3D.

Intra oral scan data, yes...CBCT data for intra oral measurements is not accurate enough for aligner production :)

Excellent! I also discovered that Autodesk has something along these lines, although as with all of these systems we need to try them to see what they can do.

Autodesk ReCap 360

You can get inspiration from the following products:

Francois-Xavier

Have you got any further with resolving this problem? I think that it is a case of converting a geographic to mathematic coordinate system, therefore if the coordinates are converted to local datum it might work, it has worked for one of my models so far but failed with a few too!!

See http://members.tripod.com/~Gensav

Hello,

As Mohamed said, your problem isn't fully specified yet. Your above example would yield different partitions if you "extend" your lines in different order.

For the intersection implementation I would recommend to use a robust library like Boost Polygon or CGAL 2D regularized boolean set operations. These are C++ libraries (where you would need to write a wrapper assembly in C++/CLI), or you can use Clipper which has a C# interface.

i highly advice you a builded workstation like the Dell Precission 7910  at max configuration

you will gain high performance and no worry about compatibility between components 

is extramly expensive but for your kind of application and there future you need this kind of computer 

I would also strongly recommend that you look at MIPAR at http://MIPAR.us. FIB SEM serial sections often require challenging segmentation, which is where MIPAR excels and many others struggle. We specifically designed the 3D aspect of MIPAR to handle FIB SEM datasets. I am happy to make a demo with a sample of your dataset if you'd like.

Cheers,

John

To inject scale in the reconstruction, you generally need to have external and/or prior information related to either of the following: a) Depth of the scene (i.e., the depth of some or all of the reconstructed points), or b) Prior knowledge or measurements of camera motion,

In robotics, the most typical way of obtaining scale is to either have a motion model (differential drive for instance) of your robot and additional sensors such as an optical encoder (accelerometers can also be used by they re not very reliable if used on their own; they can be very useful when combined with a motion model however). Once you get an estimate of the  baseline length (i.e., the difference in the robot/camera position) between two image captures, this is effectively your scale factor.

Another way of obtaining scale is by incorporating a depth sensor such as the Kinect. In this case you have reliable estimates for points that lie in the range of 0.5 - 4.5m from the camera. Provided that your scene indeed contains objects at this depth (typically indoor environments) then the Kinect is a very reliable source of scale for your Structure from Motion method.

Once you have either depth or baseline length, you would have to scale all your points by the recovered factor (there are many ways to do that, some of which are described in my PhD thesis). Regardless, you would afterwards have to perform bundle adjustment in the new estimates. If you are bothered by execution time, my suggestion would be to optimize only a few of the points that you positively regard as inliers along with the baseline vector of course.

Thank you Filipe, that is exactly what I need, I need to minimize the difference by using my PCA model and changing the ROI 3D structure.

Thank you Chao Xing and Raphael Rouveure. That was helpful.