Dr. Jiju Poovvancheri is making his Saint Mary’s University students think differently about the power of 3D sensing and virtual and augmented reality. - Photo Courtesy Mike Dembeck.
Students at Saint Mary’s University in Halifax are working on immersive, ground-breaking innovations in three-dimensional (3D) sensing and virtual and augmented reality —and they’re having fun too.
Dr. Jiju Poovvancheri, their Computing Science professor, currently has four projects set to influence industries and change entertainment dramatically, including the reconstruction of large-scale cities into “lightweight” 3D models.
“3D digital models of urban scenes can be used for city planning, emergency response training, assessment of solar energy potential, robotic navigation, entertainment, among many other applications,” says Poovvancheri, whose research focuses primarily on computer graphics and 3D computer vision.
Dr. Jiju Poovvancheri currently has four projects set to influence industries and change entertainment dramatically, including the reconstruction of large-scale cities into “lightweight” 3D models. - Photo Courtesy Mike Dembeck.
But creating these models from defect-laden 3D scans acquired via Light Detection and Ranging (LiDAR) sensors is a technical challenge, says Poovvancheri.
Drones scan buildings and other towering objects in the city from the top and sensors mounted on vehicles (such as Google Street View cars) scan point clouds (trillions of data points captured), representing all the objects. This data is then converted into a 3D model by specialized urban reconstruction algorithms.
“A point cloud is turned into a triangular mesh (for visualization and navigation purposes). But because city-scale 3D meshes contain trillions of triangles, storage is a huge concern, which makes the application slow to navigate or takes ages to render over web interfaces,” explains Poovvancheri.
Instead, he is working on a way to create lightweight 3D models that consume less storage without compromising the quality of the models.
Generated from LiDAR (Light Detection and Ranging) point clouds, these three-dimensional models are used to map urban areas. The top row is of a textured building model from Google Earth 3D, the second column: Model without textures, third column: Building model with enhanced geometric details, fourth column: Detailed models placed in Google Earth 3D. The area considered here is the downtown of Lexington, Kentucky, the U.S. And the second row are buildings generated from airborne laser scans of Toronto city dataset, left: Less detailed models (LOD1), middle: Moderately detailed (LOD 3), right: Highly detailed models (LOD5), where LOD stands for Level of Detail. - Photo Contributed.
The next frontier for Poovvancheri is the creation of volumetric videos for the benefits of face-to-face communication, like the holograms in the television series Star Trek.
“By 2025 or 2030, video conferencing communication platforms like Skype, Zoom, etc. may be replaced by holograms that recapture all my facial expressions and actions, while communicating with a person anywhere in the world,” said Poovvancheri.
To interact in real time, the remote participant will wear AR glasses such as Microsoft HoloLens glasses that superimpose a digital picture and enhance it onto a user’s view of the real world.
“One major question we are addressing in my lab is real-time compression of volumetric videos. A multi-camera system captures my actions frame by frame, where each frame consists of billions of points that represent my body pose, which is slow across the computer network when transferred. We are working on the real-time compression of the scanned data without compromising the quality of the 3D models,” he says.
AR glasses will reflect and amplify the full 360-degree breadth and complexity of the human experience. Workers can also wear AR glasses to perform tasks while viewing real-time, task-relevant information within their visual field.
“We are working on a machine learning algorithm to infer the spatial position and orientation of the digital overlays in the dynamic setting, i.e., head movements of the technician, that too in the presence of occlusions,” said Poovvancheri.
“For example, a technician can wear AR glasses to repair heavy and complex machinery with the help of digital overlays of 3D machinery models.”
Undergraduate and graduate students can apply for funding to work with Poovvancheri on these innovative projects, which “gives them great insight into the industry,” he says, adding the university administration is very supportive in undertaking such futuristic projects and encourages interdisciplinary research.
His former students have accepted employment offers at Amazon and IBM, as well as Nova Scotian companies such as Perennia. “When I see my students succeed, I feel a great sense of pride.”
“Mr. Sumesh Thakur, one of my graduate students is working on a LiDAR-based pedestrian and bicyclist detection algorithm which may be used as a part of the 3D perception system in a fully self-driving car,” he says.
“If you go through the 3D object detection accuracies on KITTI, an online standard benchmark where various companies and academic labs are continually uploading new data, you can see that the current best for pedestrian detection is 46.88 per cent on moderately difficult data,” explains Poovvancheri.
“It’s a long way to go to reach 90 or 100 per cent. We are working on a graph neural network for pedestrian and bicyclist detection.”
Pictured are qualitative 3D detection results of a graph neural network designed and developed by Dr. Poovvancheri’s group on the pedestrian and cyclists’ categories of the KITTI vision benchmark test dataset. Detected objects have red 3D bounding boxes and green 2D bounding boxes. - Photo Contributed.
The goal is to improve the accuracy statistics of the state-of-the-art in 3D object detection over the next couple of years and extend the detection system for smaller objects. e.g., trash bins or mailboxes, keeping in mind the possibility of autonomous robots for garbage collection or mail delivery.
“It comes down to 3D deep learning algorithms,” he says. “Essentially, we design deep neural networks, train the network with the labelled data available in standard benchmark suites, e.g., KITTI or Google Waymo, which eventually make the neural network capable of identifying the difference between a car versus a pedestrian or road markings. With accurate perception systems, the autonomous vehicles can help a driver avoid collisions.”
Poovvancheri is passing his enthusiasm for 3D imaging and sensing technology on to his students to help the world in more ways than one.
This story first appeared on the Chronicle Herald website.