The noise and presence of boats can harm humpback whales' ability to communicate and socialise, in some cases reducing their communication range by a factor of four.
The discovery was made by Dr Rebecca Dunlop from The University of Queensland's Cetacean Ecology and Acoustics Laboratory, who monitored the acoustics and social behaviour of humpback whales off Queensland's Peregian Beach.
"It appears that groups of humpback whales tend to socialise much less in the presence of vessels - in this study, mostly small fishing boats - compared to when there aren't any present," Dr Dunlop said.
"Before this, we didn't really know how human activity like this affected the social interactions of large whales.
"We'd previously looked at how these sounds affected whales' movements; for example, seeing how they avoided sound from seismic testing or vessels, or how their vocal level changed due to these sounds, but we hadn't studied changes in social behaviour.
"Socialising is incredibly important for migrating whales, as healthy social behaviours are closely linked to healthy breeding."
Dr Dunlop recorded land-based observations of whale socialisation, coupling this data with acoustic recordings using an offshore hydrophone array.
She compared these acoustically measured social interactions when vessels were or were not present.
The researcher said the findings might be due to the noise of the vessels masking the whales' signals.
"If they can't hear each other, they're less likely to join together," she said.
"But this only explains some, not all, of the results, as many of the signals the groups gave out should have technically been 'heard' by other whales above vessel noise.
"It's likely there's also another factor; the fact that the whales' behaviour is most likely changed by the physical presence of the vessel too."
Dr Dunlop said, despite the negative impact on the whales, the public should not be concerned that it was threatening humpback survival on Queensland's east coast.
"There isn't any evidence that our humpback whale population is under any pressure - they are increasing as rapidly as biologically possible," Dr Dunlop said.
"Therefore, any changes in breeding interactions in the presence of vessels likely has a negligible effect on the population."
She said if the population came under additional pressure from changes in krill populations, increasing human activity or other factors, then the presence of vessels might have a more negative impact.
"This would be the same concern for a less robust population - one that hasn't recovered from whaling to the same extent the Australian populations has."
The research was published in Royal Society Open Science (DOI: 10.1098/rsos.190967).