The Technology Behind Virtual Reality And Augmented Reality Applications in Clean Tech

Last time we looked at the kind of applications in clean technology where using virtual reality or augmented reality systems are making a significant difference. 

 

But, how do these systems work?

 

Most AR and VR systems can be broadly classified as follows.



These systems can be divided into 1) the hardware required to get the data, process and display it 2) the software needed to develop simulations of the systems being studied and create virtual objects and 3) the server where the data are stored and processed and where machine learning algorithms can be deployed to improve outcomes.

 

Key Hardware systems: The hardware systems can be categorized into the input systems and the display systems. 

 

Most AR and VR systems use GPS (to determine location), cameras (to obtain the live images of where the user is located and/or looking), gyroscopes and accelerometers (to determine speed and direction of the user’s movement) and other sensors that are specific to the problem being solved (a sensor that detects a chemical for example to find leaks or spills near pipelines). The first three input devices are common to most VR/AR applications and a typical smartphone’s sensors will usually suffice. Other sensors that are specific to the problem will either need to be integrated with the existing smartphone inputs or apps will need to be developed that use smartphone infrastructure to generate the required inputs. 

 

The output from the VR/AR application will need to be displayed. Commonly used display systems include headsets or head-mounted devices (like the Oculus Rift or Google Glass headsets) or smartphone screens. These devices are now widely available commercially and developer kits to create applications have also been released. One of the most widely used for applications other than gaming is ARToolkit. It’s an open-source, free toolkit and can be used to customize the application.

 

 

Modeling software: This is where software that is used to generate the simulations of the system being studied and create the virtual objects (for AR) or the virtual system (for VR) that is needed. 

 

For example, think of a VR system like the one developed at Stanford’s Human-Visual Interaction lab where the user was looking at the ocean changes in response to acidification. First, existing software needs to be used to model the ocean system itself - which is a fairly complex endeavor. Some the subsystems that need to be modeled include movement of water in the ocean, interactions between species in the ocean, interactions between a changing climate and ocean chemistry, and interactions between ocean chemistry and species living in the ocean. Additional software might need to be written to improve the simulation or add more features. Then, after the physical, chemical and biological models have been created, the simulation needs to be displayed so that the user can enjoy it. This would require creating graphics of ocean waves or coral reefs or dolphins as well as a virtual representation of the user, if needed. 

 

This is the section where expertise in the clean tech field being studied becomes critical. All aspects of the system being displayed or studied needs to be understood so that realistic models can be developed, which will then be turned into displays for the user. 

 

From a design aspect, this is the part where generating the display to make it visually appealing as well as accurate is important. A good UI designer and software engineers who are working on display are important here, if a commercial product is being developed. 

 

 

Server: This section is where the data engineering aspects of the problem come into play as well as some of the machine learning and data science aspects. 

 

The data generated from the hardware inputs, the model software all need to be stored and retrieved efficiently as the user is interacting with the system. So, in addition to the processors in the display devices (like the smartphone or gaming console), cloud storage is going to be needed. This leads to the typical issues with any data problem - how to store the data, what is the best system to retrieve the data, how can the data be processed to solve the problem? For example, if images are being received from the camera, these need to be stored and image processing algorithms written to understand and identify the images. If models of power transmission lines are being created, they need to be stored with location identifiers so that the data can be retrieved and overlaid on an existing image of a road.

 

A data scientist or data engineer would most likely be spending a significant amount of their time solving problems in this section. 

 

Any commercial application in using VR/AR to solve clean tech problems is going to need a wide range of engineers and scientists to work in all the three sections here - UI designers, software engineers, hardware engineers, data engineers, data scientists, environmental scientists, environmental engineers……

 

Of course, the most interesting aspects would be when the clean tech model needs to be integrated with imagery data or sensor data! That’s when a combination of software/data science  and clean technology skills would be powerful.

What our community are reading

Moonshots, Models, IoT and Machine Learning in Agriculture

  What do Google, Climate Corporation, early stage startups in farm robotics, and researchers trying to figure out how to feed the world sustainably have in common? They’re all grappling with one of the toughest challenges of working with natural systems - how do you work with data that is sparse, unevenly distributed and with systems that have so many connections and interactions with other systems?     Before the advent of cheap sensors that are connected to phones, easily accessible satellite data and drones that can fly over fields quickly and inexpensively -     scientists in companies and academia worked on developing plant and crop models that incorporated as many aspects of the farm and as much data as was available so that they could understand and predict what was likely to happen on the field. Understandably, the forecasts took some time to produce and as the models grew more complex, issues about how to estimate model parameters and the uncertainty a...
Read more

Our online community space is now live!

 Our online community space is now open to anyone who has signed up for a free or paid course on our website! In addition to everyone who signed up for our cohort-based courses, we're now expanding it to all the members of our community. If you've already signed up for any of our courses, check your email for the invitation for the space. It's where we'll get together to talk about all things data science and clean technology related, discuss the latest research, network and make connections with other professionals in the sector. It's an invitation only , no bots and no trolls allowed space - so come on over! Here's where you can check out our courses and join our community !
Read more

What’s the impact of your smartphone? Mapping the distribution and environmental impact of mines around the world

Image
How many different materials go into making your smartphone? According to the American Chemical Society, at least 64 different types of chemicals constitute your smartphone . Many of these materials, particularly the heavy earth metals, are extracted from the Earth through mining operations. Mining has been a part of human society for several thousand years now - from the shallow mines of earlier centuries to the more extractive ones created in the last hundred years. The challenge of mining and other extractive industries is that they are essential to many of our needs today - coal for energy, silicon for solar panels and heavy earths for electronics being among the well known ones. At the same time, they cause significant damage to local environments through the disturbance to the land caused by establishing the mine itself, to environmental impacts on water, land and ecosystems from mining operations.   So far, it’s been difficult to map the spatial extent and environmental impa...
Read more