Fifth-year engineering student Chong Di (left), Professor Jie Gong and Shengyuan Feng, a graduate student studying artificial intelligence, pose with Echo the robot in the Zimmerli Art Museum. Photo by Luca Mostello/Rutgers University

The public may soon be able to open their laptops to venture into the Zimmerli Art Museum at Rutgers University–New Brunswick and explore its galleries via an online virtual tour.

Since late August, Jie Gong, an associate professor in the Department of Civil and Environmental Engineering at the Rutgers School of Engineering, has been working on a project with Zimmerli staff to map the building. He said the mapping effort can be used for multiple applications, such as creating a virtual tour of the museum and its exhibitions (in which viewers can inspect the art online) or helping visitors who have low vision, are blind or have other conditions navigate inside and outside the museum safely.

“We want to make sure we’re accessible to all visitors,” said Maura Reilly, director of the Zimmerli. “Everyone is welcome.”

Maura Reilly, director of the Zimmerli. Photo courtesy of Zimmerli Art Museum

Gong and his team – including graduate, undergraduate and high school students – visited the Zimmerli often during the past several months to create three-dimensional models within and around the museum.

“I think this is a great STEAM partnership,” said Nicole Simpson, the associate curator of prints and drawings at the Zimmerli, referring to the science, technology, engineering, arts and mathematics aspect of the mapping effort. “It's exactly the model of how the sciences and the humanities can come together to really enrich each other and produce a project that will have a great impact.”

The Zimmerli project follows the team’s mapping of the Rutgers Geology Museum in Geology Hall on College Avenue and the SHI football Stadium in Piscataway, said Gong, adding that the plan is to map the entire Rutgers–New Brunswick campus.

He said Rutgers and a team from the City University of New York led by Zhigang Zhu, a computer science professor, have been working on a collaborative National Science Foundation project on improving accessibility of cities for people with disabilities.

“We have been a long-time partner in this line of research,” Gong said.

Gong’s mapping crew, including Rutgers graduate students Chong Di, Jiahao Xia and Shengyuan Feng, used a light detection and ranging (lidar) scanner – a device that emits pulses of laser light to determine the presence, shape and distance of objects – to capture panoramic, high color resolution 3D imaging inside (including the artwork) and outside of the building.

Photo by Luca Mostello/Rutgers University

The professor said the lidar technology maps various aspects of an area, such as the slope of the sidewalk as well as other “surprises” and obstacles that may present a challenge for people to navigate.

“The immediate application would be documenting the accessibility infrastructure in and outside of the museum, and making that information available to the public,” said Gong, adding that a long-term application would be creating assistive tours with augmented reality and virtual reality technologies.

Gong said another application of these mapping efforts might involve the university’s remote-controlled, four-legged robot designed by Boston Dynamics. The commercially available model, called Spot by the robotics company, is used at warehouses and by police departments to perform various tasks. Gong nicknamed the robot Echo.

He said Echo, which is spatially aware and able to negotiate stairs and tight corners, could be used as a museum tour guide or navigation assistant.

Two students from East Brunswick and Westfield high schools worked on Echo during their summer break, writing programs to control the robot to map building interiors and learn about the indoor space so the robot dog could localize itself in the building. Chelsea Duan, a Rutgers undergraduate student involved in the project, is developing virtual and mixed reality applications for the museum that will be used for tours, Gong said.

“We want to provide a joyful experience for users,” said Di, a fifth-year graduate student focused on civil engineering who added the team is exploring “a more efficient workflow” of scanning the museum to include new exhibitions.

Reilly, the museum’s director, said she had wanted to put together a consortium of faculty that could work on projects and bring together people interested in accessibility issues – and Gong was ready to make it happen.

“I said, ‘Please pilot with the Zimmerli because your technology aligns perfectly with our desire to be a fully accessible museum,’” Reilly said.

Reilly added she is excited about the possibilities for the Zimmerli.

“Thinking back to when we were closed during the pandemic, we tried to create a lot of virtual content with our existing resources, for audiences with varying levels of abilities, and created some more static virtual exhibitions and tours of our spaces using photographs,” Reilly said. “But it's really been extraordinary to partner with the engineering team. They have the technology and the expertise to create these more immersive and exciting experiences for our online viewers.”

Rutgers University–New Brunswick is where Rutgers, the State University of New Jersey, began more than 250 years ago. Ranked among the world’s top 60 universities, Rutgers’s flagship university is a leading public research institution and a member of the prestigious Association of American Universities.

It is home to internationally acclaimed faculty and has 12 degree-granting schools and a Division I Athletics program. It is the Big Ten Conference’s most diverse university. Through its community of teachers, scholars, artists, scientists, and healers, Rutgers is equipped as never before to transform lives.