#whales#naturedocumentary#documentary Are the sounds and acoustic signals whales make similar to human language? Biologists agree the animals communicate with one another. This film takes a journey into the depths of the oceans to decipher the language of the orcas using AI.
Killer whales live in small pods. Within these family groups they communicate with one another using their own sound dialects. With the help of artificial intelligence, scientists are now looking for repeated patterns which would indicate that orcas have a language model.
A team of researchers headed by Professor Elmar Noeth and Rachael Cheng from the University of Erlangen-Nuremberg spent three years collecting data on the movements and vocalization of various orca pods in the Pacific. Their aim was to investigate a possible connection between the patterns of the killer whales‘ calls and their behavior. Using special microphones, they captured the underwater sounds and behaviors of these marine mammals. Then they tried to determine the whales‘ position and classify their calls. It’s a first step towards solving the mysteries of whale song.
Using artificial intelligence to decipher unfamiliar communications systems is not without its difficulties. However, it’s already clear that AI will transform our ideas about highly communicative species in the coming years.
Here, off Canada's Pacific coast, researchers are hoping to make a long-held dream come true.
Behavioral biologists and IT experts have teamed up
to create programs aimed at deciphering acoustic signals from animals.
Could artificial intelligence help to identify patterns in the sounds made by marine mammals?
Will we soon start to understand what it is that whales are talking about?
The coastal waters between Alaska in the U.S. and Canada's Vancouver Island
are where a group of killer whales, or orcas, spend their summers.
A team of scientists from the Deep Owl Research Expedition are preparing to embark.
Deep Owl stands for Deep Learning Applied to Animal Linguistics.
Computer scientist Eman Nöth from Germany's University of Erlangen
has spent years working on automatic recognition for human speech patterns.
Can the same methods be adapted to animal languages?
Underwater microphones embedded in tubes serve as the expedition's ears.
For three summers, teams of computer scientists and biologists
have set out to record orca calls and document whale behavior.
Rachel Cheng from the Leibniz Institute for Zoo and Wildlife Research in Berlin
is looking for patterns between behavior and animal vocalizations
that could help to decipher whale language.
I assume they would exchange information.
They may have a very, very different system which may not fit into our prototype language.
Around 300 orcas are identified as Northern Residents
as they spend each summer along the coast of Alaska and northern British Columbia.
They live in small family groups and are very communicative.
Do different families use different dialects?
And is it possible to discern the meaning of individual calls?
Here, the researchers lower the highly sensitive microphones into the water.
The eight hydrophones can record sounds up to a frequency of 100 kilohertz.
That's far higher pitched than what is audible to the human ear.
Thanks to the network of hydrophones and acoustic triangulation,
the researchers will later be able to calculate the positions of the whales.
To avoid disturbing the whales with engine noise,
the research trimaran is equipped with an electric motor.
While visibility is limited underwater,
sound waves are transmitted over considerable distances.
That's why a communication system with loud calls is clearly beneficial.
They're very tight together and frequent direction change, circling
and body twisting, plus a lot of echolocation.
They're socializing like the U.S. Sea, they bump into each other
and rub and frequent twisting and then jump onto each other.
It sounds like they talk about the plan.
What are we going to do next?
Orcas only spend about 5% of their time at the surface,
which makes systematic observation quite difficult.
The research team uses drones to document the behavior of the animals.
Diving expeditions with whales are prohibited in Canada.
The scientists are looking for the smallest meaningful units of communication.
Which whale is calling and which one answers
Are some sounds repeated more than others?
Biologist Elisanne Durand examines the recordings.
Each call comprises a series of brief pulses,
which sound like melodic curves to human ears.
Each shift in sound could be meaningful.
While recording the whale calls,
the researchers also document the behavior of the animals.
So there's four individuals in this group here that are circling.
There were four back there with the male.
There's one that I can see out of the corner of my eye coming towards us
and the mom and calf.
So that's 8, 10, 11 individuals minimum.
The more data available, the easier it is to train
deep learning programs to decipher whale language.
It's therefore a major advantage for the researchers
that over 20,000 hours of orca calls have already been collected,
more than for any other animal species.
Whale researcher Jared Towers has been tasked by the Canadian government
with observing different orca populations.
Back in the 1970s, scientists here began documenting individual animals
as well as their group structures.
A 108 straight underneath the boat here.
There she is. She's gone off up ahead.
Towers has no problem telling the orcas apart.
So this scratch here on A 108 has persisted for at least a couple years now.
So the way that we identify the individuals,
no matter what population they belong to, is by appearance.
And you get used to looking for certain features on an individual killer whale.
The dorsal fins and the patches around the fins have different shapes,
enabling scientists to catalogue each of the northern residents.
Every family forms a lifelong bond.
What we're looking at with all of these families is an adult female leading the group.
And the fathers of their offspring don't play much of a role in that family group.
Jared Towers works for a federal institution that safeguards Canada's waters.
His former boss, John Ford,
was the first to distinguish between the disparate calls of the whales.
Ford's research revolutionized our understanding
of the communication system used by the resident orcas.
What we're listening to are calls of A clan whales.
That's the exciting part of underwater listening,
is you're getting a window into their life that you would never see.
John Ford discovered that the whales use about 50 different calls.
Different families prefer different types of calls.
These were nicknamed dialects and used to help identify individual families.
When they're making the various stereotype signals,
it's simply to keep in touch with everybody in the group.
They exchange them.
They're constantly monitoring each other's location
because they can tell where they are because of their very directional hearing.
And they can monitor the sort of behavioral state,
the excitement level, the arousal state of all the other animals in the kin group.
Northern resident orcas visit pebble beaches daily.
You can hear them rubbing right now.
They're making socializing sounds,
and you can hear the pebbles getting pushed around.
And they just rub all sides of their body,
just under the in the shallow part of the beach.
Only a few orca groups worldwide engage in this sort of body rub.
This behavior is not genetic.
Rather, it's a tradition passed on within families of the northern residents,
just like their language.
Back on the German-Canadian research boat,
it's a challenge to locate each family
among the 300 individuals that make up the northern resident population.
The orcas are constantly on the move in an expanse of water the size of Belgium.
The expedition covers an area from Vancouver Island to the southern tip of Alaska.
Seagulls indicate where schools of salmon might be.
And this is where orcas often hunt, too.
Calls from the A clan can be heard on the underwater microphones.
The team tries to determine the position of the whales.
They compare the calls with a catalog compiled by John Ford,
but they encounter discrepancies.
The calls of class N9 are used by different whale families.
Calls from the same class should be almost identical,
but these differ in length, melody, and harmonics.
Human analysis so far has amounted to only a rough classification.
Looking at the spectrograms, I'm very sure that we can,
you know, achieve something that rivals the human performance.
Two whale families approach.
What calls are they exchanging?
It was that call.
Okay, just to record this.
After I hear the call, the back group surfaced,
and then I spotted the front group turn around.
The researchers are interested in which group is calling,
which one answers, and which sounds they're using.
So look here, the I4 is approaching the boat,
and you can hear a lot of calls.
And here we can see the calls they make.
Oh yes, it's N23.
So there's a lot of variations also.
And interestingly, here is the A23 families.
The calls look really different.
Yeah, it's how people differentiate different matter.
If we have a lot of calls from them, we can try to train a classifier.
The programmers use algorithms, or classifiers,
to automatically analyze millions of whale calls
in order to compare recurring sound patterns with recurring behavioral patterns.
This same method is used to decipher the meaning of individual words in foreign languages.
For Elmar Noet and his team,
it's no easy feat to automatically filter out the weak orca calls
from the constantly fluctuating noise of the ocean.
By matching thousands of orca calls on the input side
with a target value on the output side, they train neural networks.
These layers of neural networks then learn, bit by bit,
which sounds are typical for orcas.
The purple peaks indicate a high probability of having found a whale call.
It's the first time a machine has been trained to automatically detect orca sounds.
Still, clustering calls into similar sound groups remains a big challenge.
When comparing tens of thousands of recordings,
all whale calls are sorted by what they have in common.
But the cluster is still too crude to form reliable classes.
Further programming is required.
The team suspects that the northern resident orcas have a complex language system
and may even process more intricate sound patterns than humans.
But how much is even known about the acoustic capacity of their brains?
This is Telegraph Cove on Vancouver Island,
where Elma Noet and Rachel Cheng have an appointment.
Only a handful of researchers worldwide have expertise on whale and dolphin brains.
Neuroscientist Lori Marino is one of them.
Is this an orca brain?
This is an orca skull, a cranium, right, of an orca.
And in the front, you would find the melon, where they do the, where they direct the echolocation.
And this part right here, this large part is where the brain would sit.
Wow.
Okay.
Yeah, yeah.
Human and whale brains generally share the same mammalian architecture.
Both cerebral cortices are wrinkled and complex.
But the orca brain is five times bigger, among the largest of any animal.
And unlike the human brain, the orca also has a paralimbic lobe.
The part of the orca brain that really fascinates me is this part right here.
We have this special extra lobe called the paralimbic lobe.
So it connects feelings, emotions, and thought.
And it is so complex in the orca brain, it has become its own lobe.
Okay.
So that could mean that a lot more of the communication has to do with their feelings,
with their emotions.
That communicating feelings, communicating emotions are very important to an orca.
Okay.
Whales have a far larger acoustic area than humans.
Lori Marino's team also discovered a second acoustic cortex.
Now we know that they also have a sound processing area in the same area that we process language.
Based on what you said about how they process sound,
can we say that we need a much finer window when we analyze?
So it's not just fine resolution that we need.
We need a way to actually do the analysis at a much more complex level than our brains can do it.
New brain scan methods have revealed that orcas
have a far more complex acoustic system than previously believed.
Could the same be true for other marine mammals
that use sound for their communication and orientation?
In contrast to other orca groups,
the northern residents at some point stopped hunting seals and other marine mammals.
Today, they feed almost exclusively on the largest salmon in Canadian waters, the Chinook.
Stocks of the fish, however, have been declining rapidly since the late 1990s.
The northern residents have to travel increasingly long distances
to find the salmon that still remains.
And despite stringent fishing quotas for the species, their numbers continue to decrease.
Jared Towers from Fisheries and Oceans Canada
monitors if people are complying with the fishing regulations.
It seemed to be that when Chinook salmon aren't quite so abundant,
the killer whales aren't getting enough to eat,
and that causes the population to go down a little bit.
The northern resident orcas are classified as a threatened species.
When Chinook salmon abundance is low, they don't simply go out and try catching other fish.
They haven't learned or they haven't found an ulterior prey resource.
It's not like they're going to go out and start killing harbour seals
just when there's not enough salmon around.
The substantial human appetite for salmon has decimated stocks to critically low levels.
The ruins of this salmon fishing industry
serve as a reminder of the once abundant fish stocks.
Wild Canadian salmon were in great demand around the world.
The survival of the northern resident orcas is tightly linked to the fate of the Chinook salmon.
Lance Barrett-Leonard is a geneticist and behavioural ecologist at the Vancouver Aquarium.
He visits the resident orcas every summer to document their long-term health
and measure how their body size and shape compares to previous years.
To ensure the images are comparable from year to year,
the drone hovers exactly 30 metres above the whales.
The group's calf is almost a year old and is still suckling from its mother.
The social skills of orcas are illustrated by how they deal with their prey.
You've got a species that's highly social, that shares all of its food.
So they're compulsive sharers, killer whales.
So if they catch a fish, they'll, you know,
one whale will eat half and hand it off to somebody else and they'll bite him
and then another one will come in and take a little bite.
The adults sometimes leave living salmon to the younger whales.
In this way, the juveniles learn to hunt their prey in a playful way.
At the Vancouver Aquarium, researchers compare the images from different years.
One indicator of the whales' state of health are the white patches above their eyes.
If there are not enough fat reserves in the neck area, the angle along the eye spots will change.
Comparing the images side by side, researchers observe a life-threatening weight loss.
In response to the decline of natural stocks,
the fishing industry installed aqua farms in the regions the remaining wild salmon migrate through.
Nets protect the farmed salmon from orcas and other predators.
The predominantly Norwegian operators introduced farmed Atlantic salmon species
and with them new viral diseases not previously found in Canada's Pacific Northwest.
These fish farms raise as much salmon as possible in a confined space.
It's all about maximizing profit.
The byproduct? Feces and germs that escape into open waters.
There is growing concern of viral outbreaks among the wild Chinook salmon.
If the primary food source of the resident orca population continues to decline,
the whales' own survival will also be endangered.
Whale researcher Alexandra Morton is studying the new diseases affecting the salmon.
She visits the research trimaran to share the results of her analysis.
I brought some results on my laptop if you'd like to see them.
Of the virus research.
Where we found it and why are the salmon farms such a big problem for the salmon and the whales.
Mostly because of where they're located.
So we do a molecular test which basically just looks for the genetic sequence or part
of the genetic sequence of the virus.
Salmon take in viruses through their gills.
The pathogens then infect their bloodstream, causing organ failure.
When the salmon farms moved in, the orca stayed away.
That made Alexandra Morton suspicious.
She took samples near the farms, which were then analyzed in special laboratories.
She examined the organs of both wild and farmed salmon.
95% of the farmed fish she dissected were sick.
A large proportion of the wild salmon also became infected through contaminated water.
The pathogens escape from the fish farms with the tidal currents
and spread to the regions the young wild salmon have to pass through as they migrate.
200 kilometers further north, the A42 family searches for prey along the shorelines.
The research expedition has reached the fjords of Fisher Channel,
once famous for its ample fishing grounds.
The whales hunt by day and by night.
They don't need light for orientation.
Biologist Florence Sullivan compares her recordings to known call types and notes down behavior.
Do the orcas tend to use certain calls in a social context or more when hunting?
All recordings are later reanalyzed in Germany with the help of deep learning programs.
In their search for prey, the northern resident orcas now often cover 80 to 120 kilometers a day
along hunting routes that have been passed on for generations.
To identify individual whales and examine changes in group structures,
programmer Manuel Schmidt and biologist James Field are working on automatic fin recognition
software. It works similarly to facial recognition programs. The process of identifying whales by
their fins and dialect was used years ago in the famous case of an orphaned calf known as
A73 or Springer. When her mother died, the then two-year-old Springer was separated from the rest
of her family. Eventually, 300 miles further south, the lone calf was found off the coast of Seattle.
Scientists could tell from her calls that she was a northern resident.
Springer was brought back and reunited with her family.
So while researchers have been able to identify whale dialects for 20 years now,
understanding the language of whales represents a whole new challenge.
The call systems used by different whale families change little from one generation to the next.
Springer now has two calves of her own, Spirit and Storm, roughly eight and four years old.
Orcas typically spend their entire lives with their mothers. For the first two years,
they are suckled and over the following eight years, they learn what they need to do to survive.
Another orca group has joined Springer's family. Together, they search for salmon close to the
shore. Their hunting strategies vary depending on the local environment.
This gives them a wealth of experience that, like their communication system,
is passed down through generations. Their group behavior during the hunt
seems to be coordinated. The whales exchange a stream of calls.
The orcas are still able to find enough food by spending more and more time hunting. For a calf,
it's one of many lessons in salmon hunting. For the scientists, it's a rich source of data.
Human activity greatly impacts the whales' habitat.
The sound of passing ships doesn't frighten the whales,
but the noise does interfere with their communication and compromises their hunting
ability. The Canadian government is investigating the impact of ship noise on marine mammals.
So we listen for their vocalizations, which can cover a range of 10 kilometers or so.
Man-made noise in the world's oceans doubles every 10 years.
One of the concerns we have about vessel noise is its potential effect on the echolocation
capabilities of the whales. The whales' vision is very limited underwater because of murkiness in
the water, especially at night. They can't see. Typically, in these waters, more than a whale's
length or so, so perhaps 10 meters max. In quiet conditions, the whales can probably detect a
salmon at a range of perhaps 200 meters, but that might be significantly reduced by
masking of by boat noise. Similar to bats, orcas scan their surroundings by emitting clicking
sounds and listening to their echoes. The noise from ships masks those echoes. To reduce its own
engine noise, the expedition boat runs on an ultra-quiet electric drive.
It sounds like an M44, but it's masked in the noise. Here you have the boat's engine noise.
And there's an orca coming. So I have another recording of our engine.
You can hear the other harmonics. It's much clearer, but still we have this electrical
noise here. 10 kilohertz. Let me have a look at the engine. Yeah.
Although electric engines are far quieter, there's nevertheless some interference.
Rachel Cheng inspects the static noise and examines the control cables.
Is it better now? Much better now.
The sound of electric motors is still rare in the ocean,
which is perhaps why the ship triggers the whale's interest.
The boat stops in order to maintain the requisite distance of 100 meters.
The curious orcas are not as cautious and inspect the boat and its crew.
The scientists wonder whether the whales might use distinct calls for different boats.
In any case, there is immense curiosity on both sides.
Orcas are extremely social creatures and love to touch one another.
I think that's a 23 and a 25 group floating sideways,
putting its fin on the water's surface while others just mingling around.
Are their jumps also part of their communication or just a bit of fun?
Many of their behavioral patterns are still only partially understood.
This is also due to the fact that orcas rarely roam close to the surface
where humans can observe them.
Some distance away, another group of orcas suddenly appears.
Marine biologist Elisanne Durand wants to get an overview.
OK, which direction should I go now?
If we could have the whole group on the starboard side.
This is a completely different type of orca.
There are four transients, maybe off the island.
Unlike the resident whales, these transient orcas don't feed on salmon.
They hunt other marine mammals.
Are you hearing anything from these guys, Rachel?
No, I don't hear any vocalization.
But it seems they're hunting, they're foraging.
In another contrast to northern residents, transient whales hunt almost silently.
That is because dolphins, seals and other whales, their natural prey,
have excellent hearing.
That means they know exactly when they're in danger.
The transient orcas also inspect a rocky plateau for prey.
It's a tactic that appears to be successful.
Oh, we hear them vocalize now.
The blood at the surface reveals a kill.
The transients begin to celebrate their catch and now exchange many calls back and forth.
Acoustically, they're very different from the residents.
I'm curious if the classifier can actually pick up the difference.
The homes of the roughly 500 transients and 300 northern residents overlap.
But their calls differ so much, it's as if they're speaking different languages.
Dolphins and other marine mammals can tell the difference.
While they immediately flee from transient whales,
they rush intentionally toward the residents,
as soon as they hear their calls, to hunt and play with them.
Below deck, Elmar Noet and Christian Bergler work on the automatic call classification.
Here, you hear the orcas?
And it shows here the detection of those signals.
Yeah, yeah, yeah.
So you always have two seconds, two seconds, two seconds.
You can clearly see that the incoming sounds, if there is an orca in it,
they will be detected quite...
What I like is that the confidence is so high,
so the algorithm is quite sure that it did find.
Yes, yes.
The automatic call detection works.
But the challenge is still to match one group of calls with another.
In human language, the equivalent of these sound sequences might be simple statements.
The northern residents consist of 32 families.
They typically use different call systems,
but are still able to communicate with each other when they meet up.
Looks like the other group is joining them.
The calls between the groups go back and forth,
announcing their arrival in a kind of greeting ceremony.
Do you still think they're saying, I'm here, I'm here, if they're so close by?
No, it's not the same call.
The last time, it was really one call that the individual was repeating all the time,
and it was the same one.
Here, they're doing like more combination.
Meetings between family groups used to be more frequent,
when salmon stocks were more abundant and they had to hunt less.
Seems to be a lot more melodic.
Their sounds are a lot, or the vocalizations are a lot longer.
I would say they're more complex.
They exchange another series of calls before the meeting slowly breaks up.
These sounds might signify goodbye.
But the researchers will need to compare many other similar exchanges
before they can interpret it more decisively.
There are thick banks of fog between Vancouver Island and the mainland
as the end of summer approaches.
Tracking down the whales is more of a challenge.
The researchers' only option is to use acoustic localization with the hydrophones.
Under calm sea conditions, the sound of the orca calls can range up to 10 kilometers.
Emma Noeth checks the location forecasts.
Localization is also important in order to interpret the calls.
It's like two or three here, two or three there,
and then it's always the same call what we hear right now.
So it's probably I'm here and the other one recognizes this one.
Oh, Fred is over there.
What I would do is try to cluster exactly those calls
where I'm convinced that I heard the same.
And then just from the sequence, one could make the assumption
that's a signal and response signal, signal response signal, something like that.
The last of three expeditions is coming to an end
with data from a total of 20 weeks of field research on board.
A few months later, the researchers meet at the Pattern Recognition Lab
at the University of Erlangen in Nuremberg, Germany.
Rachel Cheng has an appointment with the programming team.
Manuel Schmidt has run an automatic comparison of one million calls.
Hi, Rachel.
Good that you are coming.
Take a look at the clustering.
I made some changes and it's a little bit better than before.
N3 is here.
They look very similar, but it's mixed with N9 here.
This one looks very different.
But you still need training to improve it.
Background noises hinder the automatic sorting of various calls.
A newly programmed noise filter increases the precision.
Is that the new clustering?
Yes.
That's a lot better, doesn't it?
Yeah.
And it's tiny differences.
There are some tiny artifacts.
There are still small errors.
But on such a large scale, where millions of calls are being automatically compared,
they become less significant.
I think we're at the level where we can run through a couple of tapes now, several tapes.
But you can use those cluster sequences in order to find those language patterns?
Like a semantic structure?
And those can be interpreted as a call followed by an answer followed by another call.
The new clustering results are better than expected.
Rachel Cheng's task is now to assign recurring call patterns to certain behaviors.
She compares the new clusters with the existing call catalogs
and discovers reappearing matches that might be meaningful.
The algorithms have learned to differentiate the calls of different orca families.
That would normally take human researchers years of training.
In the future, this will allow automatic detection of which family is calling
and the ability to follow the subsequent dialogues between families.
The next step is comparing millions of call sequences.
Which calls appear together frequently, accompanied by which similar behavior?
When decoding human languages, such connections might reveal meaningful
possibilities such as sit and chair or table and plate.
The deep learning programs cannot yet work completely independently.
Without human control, there could be a huge increase in assignment errors.
The first matches appear.
The same call sequences were used in a similar context.
But there are exceptions.
The comparisons made so far are not conclusive.
So the reappearing call sequence point us to this scene.
It's socializing.
And we also find in the second time this in this socializing context
with the same pattern, the same call pattern.
We expect to find it in the third one also.
That doesn't disprove it.
I mean, there could be different explanations.
We just don't have enough data to say that.
The German-Canadian research team is not yet able to create
a kind of dictionary of the Orca language.
But now they do have tools to compare whale calls
in a more detailed and systematic way than was previously possible.
And as more recordings of whale calls are shared
and made available for training deep learning machines,
the faster it will be to recognize the subtleties in communication patterns.
Humans have long faced limitations when it comes to
understanding what animals might be talking about.
But with the help of AI, a new era of research
might allow us to decipher the secrets behind their communication.
you