The number 1 in the world went 6: 2: 7: 6 to Novak Djokovic when Serbia won the first team event 2: 1.
The Spaniard seemed upset when he dropped 5-2 in the first set and spoke to the referee before showing a sarcastic thumb to a loud part of the crowd.
"Honestly, the audience was fantastic every day, but sometimes people from some countries may not understand how tennis works," Nadal told reporters after the defeat.
"They think it's (how) soccer and such and the atmosphere in tennis is different. The respect for the players should be there and in some places I think that the respect of a small part of the audience is (not) there. "
Spaniard Roberto Bautista Agut had defeated Dusan Lajovic in two sets before Djokovic's victory equaled the tie.
Djokovic triumphed alongside Viktor Troicki in a double against Feliciano Lopez and Pablo Carreno Busta and secured the title for Serbia.
At the ATP Cup, which introduced a second team event in addition to the Davis Cup in the tennis calendar, 24 countries competed against each other in three Australian cities – Brisbane, Perth and Sydney – over ten days.
In contrast, the Davis Cup started qualifying rounds in February last year before ending on November 30 when Spain defeated Canada.
But Nadal said more work needs to be done to prevent the tournaments from happening every two months.
"We need to find a way to make a deal (between) the ITF and ATP to create a big team World Cup competition, not two World Cups (within two months)," said Nadal.
"I think this is confusing for the audience and we have to be clear about our sport."
The tournament swing in Australia culminates in the Australian Open, which will take place on January 20 in Melbourne.