在透射電鏡斷層成像技術(shù)中，由于電子對(duì)材料的穿透十分有限，材料在高角度的投影數(shù)據(jù)通常無(wú)法獲取。而這些缺失的投影數(shù)據(jù)會(huì)在最終的三維成像結(jié)果中引發(fā)楔形失真。

Fig. 1 Schematics of missing wedge artifact in the conventional electron tomography and our UsiNet workflow.

該研究提出了一種基于深度學(xué)習(xí)的算法，利用卷積神經(jīng)網(wǎng)絡(luò)對(duì)缺失的投影數(shù)據(jù)進(jìn)行補(bǔ)全，以消除透射電鏡斷層成像中的楔形失真。在同類研究中，由于透射電鏡斷層成像數(shù)據(jù)集的缺乏，標(biāo)簽的獲取一直是神經(jīng)網(wǎng)絡(luò)訓(xùn)練的難題。

來(lái)自美國(guó)伊利諾伊大學(xué)香檳分校材料科學(xué)與工程系的陳倩教授團(tuán)隊(duì)，設(shè)計(jì)了無(wú)監(jiān)督式的神經(jīng)網(wǎng)絡(luò)訓(xùn)練流程來(lái)避免訓(xùn)練集標(biāo)簽獲取的難題。相比于同類算法，該研究的亮點(diǎn)在于其完全不依賴于任何數(shù)據(jù)庫(kù)以及計(jì)算機(jī)模擬數(shù)據(jù)就能完成對(duì)模型的訓(xùn)練和對(duì)缺失投影的補(bǔ)全。

作者首先使用計(jì)算機(jī)模擬的納米粒子電鏡投影數(shù)據(jù)對(duì)算法進(jìn)行了驗(yàn)證，得到了對(duì)算法性能在不同的缺失投影角度范圍下，不同噪聲影響下和不同納米粒子形貌下的定量表征。

Fig. 4 Unsupervised sinogram inpainting implemented on 3D images.?

隨后該算法被應(yīng)用于實(shí)驗(yàn)上獲得的真實(shí)數(shù)據(jù)并成功消除了實(shí)驗(yàn)數(shù)據(jù)中的楔形失真。相比于其他的傳統(tǒng)校正算法，該算法能對(duì)楔形失真進(jìn)行最徹底的去除，并還原樣品中納米粒子的真實(shí)三維形貌。

Fig. 5 Orientation-dependent missing wedge artifact and comparison between different reconstruction algorithms.

該研究是基于人工智能的圖像算法在電子顯微鏡中的一次成功應(yīng)用，也是納米材料三維表征技術(shù)的重大進(jìn)展。相關(guān)論文近期發(fā)布于npj?Computational Materials?10:?28?(2024)。

Fig. 6 Comparison of 3D reconstructions of experimentally synthesized NPs with and without inpainting.

Editorial Summary

In electron tomography, due to the limited penetration of electrons into materials, projection data at high angles is often inaccessible. These missing projection data can cause wedge distortion in the final three-dimensional imaging result. This study proposes a deep learning-based algorithm that utilizes convolutional neural networks to complete the missing projection data, thereby eliminating wedge distortion in electron tomography. In relevant studies, due to the lack of electron tomography datasets, obtaining labels has always been a challenge for such neural network training.?

Fig. 7 Visualizing the heterogeneity of experimentally synthesized NPs.

Professor Qian Chen’s team from the Department of Materials Science and Engineering at the University of Illinois at Urbana-Champaign has developed an unsupervised neural network training workflow to bypass the difficulty of obtaining training set labels. The highlight of this study, compared to other algorithms, is that it can train and apply the model without relying on any databases or computer-simulated data. The authors first validated the algorithm using computer-simulated electron microscopy projection data of nanoparticles and obtained quantitative evaluation of the algorithm performance under different ranges of missing projection angles, noise levels, and nanoparticle morphologies. The algorithm was then applied to experimentally obtained data and successfully eliminated wedge distortion in the experimental data. Compared to other traditional correction algorithms, this algorithm can thoroughly remove wedge distortion and restore the true three-dimensional morphology of nanoparticles. This study represents a successful application of artificial intelligence-based image algorithms in electron microscopy and a significant advancement in three-dimensional characterization techniques for nanomaterials.This?article was recently?published in?npj?Computational Materials?10:?28?(2024).

原文Abstract及其翻譯

No ground truth needed: unsupervised sinogram inpainting for nanoparticle electron tomography (UsiNet) to correct missing wedges (利用無(wú)監(jiān)督式圖像修補(bǔ)進(jìn)行的透射電鏡斷層成像失真校正)

Lehan Yao,?Zhiheng Lyu, Jiahui Li &?Qian Chen

Abstract Complex natural and synthetic materials, such as subcellular organelles, device architectures in integrated circuits, and alloys with microstructural domains, require characterization methods that can investigate the morphology and physical properties of these materials in three dimensions (3D). Electron tomography has unparalleled (sub-)nm resolution in imaging 3D morphology of a material, critical for charting a relationship among synthesis, morphology, and performance. However, electron tomography has long suffered from an experimentally unavoidable missing wedge effect, which leads to undesirable and sometimes extensive distortion in the final reconstruction. Here we develop and demonstrate Unsupervised Sinogram Inpainting for Nanoparticle Electron Tomography (UsiNet) to correct missing wedges. UsiNet is the first sinogram inpainting method that can be realistically used for experimental electron tomography by circumventing the need for ground truth. We quantify its high performance using simulated electron tomography of nanoparticles (NPs). We then apply UsiNet to experimental tomographs, where >100 decahedral NPs and vastly different byproduct NPs are simultaneously reconstructed without missing wedge distortion. The reconstructed NPs are sorted based on their 3D shapes to understand the growth mechanism. Our work presents UsiNet as a potent tool to advance electron tomography, especially for heterogeneous samples and tomography datasets with large missing wedges, e.g. collected for beam sensitive materials or during temporally-resolved in-situ imaging.

摘要復(fù)雜的自然或合成材料，如亞細(xì)胞器、集成電路中的器件結(jié)構(gòu)和具有微結(jié)構(gòu)域的合金，需要三維的表征方法來(lái)研究其形態(tài)和物理性質(zhì)。透射電鏡斷層成像技術(shù)在表征材料的三維形態(tài)方面具有超高的納米級(jí)分辨率，這對(duì)于建立材料合成、材料形態(tài)與材料性能三者之間的關(guān)系至關(guān)重要。然而，透射電鏡斷層成像技術(shù)長(zhǎng)期以來(lái)一直受到實(shí)驗(yàn)上不可避免的缺失楔形效應(yīng)的困擾，這導(dǎo)致最終的三維重建中經(jīng)常出現(xiàn)圖像失真。本研究中我們開(kāi)發(fā)了基于深度學(xué)習(xí)圖像修補(bǔ)進(jìn)行的、用于納米粒子的透射電鏡斷層成像失真校正算法，且本研究是相關(guān)領(lǐng)域第一個(gè)無(wú)需訓(xùn)練標(biāo)簽就能進(jìn)行的基于正弦圖補(bǔ)全的失真校正算法。本文中我們首先使用計(jì)算機(jī)模擬的納米粒子斷層成像數(shù)據(jù)對(duì)算法進(jìn)行性能驗(yàn)證，然后將此算法應(yīng)用于實(shí)驗(yàn)中獲得的成像數(shù)據(jù)，并取得了良好結(jié)果。我們使用本算法在對(duì)上百個(gè)十面體納米粒子進(jìn)行的三維重建中成功避免了楔形失真，并使用這些三維表征數(shù)據(jù)對(duì)納米粒子的形貌進(jìn)行了分析，從而取得了對(duì)其生長(zhǎng)機(jī)理的了解。本工作是對(duì)透射電鏡斷層成像技術(shù)的一項(xiàng)重大推進(jìn)，也是對(duì)異質(zhì)材料、電子束敏感材料和原位成像等特殊樣品的三維表征的解決方案。