Dr. Nimisha T M
Dr. Nimisha Thekke Madam
Postdoc Researcher at TU Berlin
Research : Research : Publications : Awards and Recognitions : Activities

I am a PostDoc researcher at TU Berlin in RSiM group under Prof. Begum Demir. Prior to joining postdoc I worked as an Algorithm Development Engineer with Ebeam division KLA+, Chennai, India (Jan 2019-Jan 2021). I worked on SEM image restoration (denoising, SR), classification and detection frameworks at KLA+. I did my PhD from Image Processing and Computer Vision lab under the supervision of Prof. A N Rajagopalan, IIT Madras. My thesis addressed the problems of hand-held imaging (mainly resolution and motion blur) and proposed several multi and single image approaches to overcome it. I did my Masters from NIT Calicut in Signal Processing (2011-13), where I worked with Dr. G Abhilash in Compressed sensing based signal recovery techniques. I am a passionate researcher and have worked on multiple image restoration, detection and classification tasks covering traditional image processing methods to to the latest deep learning techniques. Check out my CV here and links to some of the academics pages ORCID iD icon

Research Topics

During my Ph.D., I worked in area of low-level vision and image restoration. My research interest includes compressed sensing, Depth from defocus, Dictionary based reconstruction and Camera mapping. I am also interested in the of-late deep learning frameworks for vision.

From videos to Pan Photography

We synthesize pan photos from motion blurred videos. Pan photography is an artistic photography intended to capture motion in images. It improves the aesthetic feel of an image. But capturing such images require great amount of skill and effort. We ease this by synthesising the same from a captured video.

Dictionary Based 3D scene Reconstruction

Sparse representations has found great application in image processing community. The central idea here is that any natural signal can be represented sparsely in an overcomplete dictionary. We use this idea to estimate the latent image and depth map from a space variantly blurred image.

Cross Camera Mapping

The photometric properties of a scene changes with varying camera and illumination. Finding a representation inavariant to these changes is of great importance in finding changes between scenes taken under different tiimes of day with different cameras.

Under-water Color Correction

Haze and color loss are the major problems in underwater imaging. When light propagates inside the water medium due to scattering particles, different wavelengths gets attenuated differently with depth. This leads to color loss and hazy affect in the captured underwater scenes. We propose here a method to color correct these images and produce its equivalent as seen from above water surface.

Blind-Superresolution of 3D scenes

Estimating depth map and a high resolution image from a bunch of low resolution motion blurred images is dealt here. Given the LR frames we estimate the HR camera motion. This is used to iteratively solve for the depth map and HR clean frame using a cost function relating the two.

Blur-Invariant Deblurring

Single image blind-deblurring is a highly ill-posed problem. We propose a deep network which can learn blur-invariant features. Our network consists of two stages. Stage I is an encoder-decoder that learns clean data representation and Stage II consists of a Generative Adversarial Network (GAN) that learns to map a blurred frame to the clean representation.

Unsupervised Deblurring

Deep networks usually work with paired input-output samples. But in most of the cases capturing such data pairs is difficult or impossible. Hence, a network that can be trained with no data pairing is what we look forward for. We propose a GAN based network for class-specific deblurring. To constrain the solution space, we also add a reblurring and gradient modules to the network.

Images confidential

SEM image denoising

SEM images are innately noisy and getting a ground truth clean data to train a supervised deep learning denoiser is not possible. Noise-to-Noise denoisers exist that works irrespective of no clean GT data. We use the same model in SEM denoising with additional cost functions tailored to SEM data.

Images confidential

SEM image Super-resolution

Collecting high resolution SEM images is time taking and might even lea to burning of the images site. To avoid this, we propose a SEM super-resolver that improves the throughput of the system without impacting the detection and classification accuracies.

Images confidential

Defect Detection and Classification

Early detection and classification of defects in chips helps in improving the yield of semiconductor industries. A detector trained on a certain field of view (FOV) struggles to give results when the FOV during inference stage. A study on how to combat this is done. Similarly, in defect classification its important to quantify the data requirement, cross-class correlations etc to understand the performance of deep learning classifier. Research on these directions were carried out.

Publications
  • Nimisha T M, Amitoz D, Raj Kuppa, and Bingxi Li "A Deep Learning (DL)-based SEM Denoiser from Noisy Data Pairs", Global Engineering Conference KLA+ 2020
  • Nimisha T M, Amioz D, and Raj Kuppa, "Learning SEM denoiser from Noisy data pairs",in Neoterix Conference 2019 conducted by KLA+ India
  • Subeesh Vasu, T. M. Nimisha, and A N Rajagopalan, "Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network," Perceptual Image Restoration and Manipulation (PIRM) Workshop and Challenge European Conference on Computer Vision Workshops (ECCVW 2018), Munich, Germany, September 2018. [Paper link] [Poster] [Project page]
  • T. M. Nimisha, Vijay Rengarajan, and A. N. Rajagopalan, "Semi-supervised Learning of Camera Motion from a Blurred Image," Accepted for publication at IEEE International Conference on Image Processing (ICIP), Athens, Greece, October 2018. [Paper link] [Project page] [Presentation]
  • T. M. Nimisha, Sunil Kumar, and A N Rajagopalan, "Unsupervised Class-Specific Deblurring," Accepted for publication in the European Conference on Computer Vision (ECCV), Munich, Germany, September 2018. [Paper] [Poster]
  • T. M. Nimisha, A. N. Rajagopalan, and R. Aravind, "Generating High Quality Pan-Shots from Motion Blurred Videos", Accepted for publication at Computer Vision and Image Understanding (CVIU), 2018. [Paper link] [Supplementary link]
  • T.M Nimisha, Akash Kumar Singh, and A.N.Rajagopalan, "Blur-Invariant Deep Learning for Blind Deblurring," IEEE International Conference on Computer Vision (ICCV), Venice, Italy, October 2017[Paper] [Poster] [Supplementary]
  • Abhijith Punnappurath, T. M. Nimisha, and A.N. Rajagopalan,"Multi-image blind super-resolution of 3D scenes,"IEEE Transactions on Image Processing., Vol. 26, No. 11, pp. 5337-5352, November 2017.[Paper] [Supplementary]
  • Nimisha T M, Arun M and Rajagopalan A.N. "Dictionary Replacement for Single Image Restoration of 3D Scenes." BMVC 2016. [Paper] [Poster] [Supplementary] [Extended Abstract]
  • Nimisha T M, Karthik S and Rajagopalan A N. "Color Restoration in Turbid Medium." ICVGIP 2016 [Paper] [Supplementary]
  • Nimisha T M, Rajagopalan A. N., and Rangarajan Aravind. "Seamless Change Detection and Mosaicing for Aerial Imagery." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2015. [Paper] [Poster]
Teaching Assistant
  • EE1100 - Basic Electrical Engineering
  • EE5175 - Image Signal Processing
  • EE5410 : Introduction to DSP
  • Basic probability
  • EE6132 : Advanced Topics in Signal Processing (Deep learning for image processing)
Awards and Recognitions
  • Recipient of Institute Research (IR) Award for Even-Semester 2018
  • My team (IPCV-team) bagged first, second and third place in track A, B and C, respectively of the PIRM Challenge (Perceptual Image Restoration and Manipulation) conducted by ECCV 2018.
  • Patent on "SYSTEM AND METHOD FOR GENERATING PAN SHOTS FROM VIDEOS"
  • Won first prize in "Code to Optimize" event conducted in the technical event Shaastra 2016, IIT Madras
  • Received travel grant from Microsoft and ACM India to present my work in ICCV 2017
  • Invited guest speaker at National Institute of Technology Calicut as a part of IEEE Signal Processing Society (Date 27/12/2019). Topic of my talk was "IMAGE RESTORATION: FROM TRADITIONAL METHODS TO DEEP LEARNING". [images1] [images2]
Activities
  • All natural signals can be represented sparsely in compressible domains. This paves the way for compressed sensing which states that data can be recovered from a sampling rate much smaller that the Nyquist rates. Check out my blog post for further details.
  • I like photography and writing the technical details on it. [Photography Write ups]
  • Though not a professional in paintings, I like drawing and painting with water/oil/arcylic paints. Check out some of my works here [link for paintings]
Top