Around Sound

Alright, whenever you're ready. Hi. We're at Poon Typic and our project is called Around Sound. So yeah, idea is basically, what we wanted to do was to help visually impaired users navigate through their surroundings using sound. And the inspiration was echolocation, so like in animals like bats, dolphins, they send out a sound and then based on the reflection of that they figure out what the surrounding looks like. So that was kind of the idea. And the way we wanted to do that was have a sound to distance mapping. So basically the device will kind of map the environment and then produce appropriate sounds to that. And most importantly have a 3D sound. So basically if the user is just wearing the device, they hear a sound coming from a direction and that's what, like they know that the object in that direction is a certain distance away. So that this way they can figure out. And so the goal was basically for the device to be able to map and which would need a sweep in a way so that it kind of goes over the entire surrounding range. And then convert all of those distances to sounds and which are distinct sounds so that the user can easily figure out that yeah, okay, this sound is from an object that's farther away than compared to this object. So the audio that you're generating is indicating the direction to the object and the distance to the object. And the direction is indicated by some sort of a head transfer function that makes it sound as if the audio is coming from the direction of the object. Exactly. Okay. Yep, exactly. So as part of the demo, we have this stick that any user would be able to use to kind of set up the device. On top of it we have a servo mounted that does an angular sweep over minus 60 to plus 60. Okay. Which pretty much maps the immediate field of vision that humans have. And yeah, the servo is actuated in a series of seven distinct sub ranges. And for each sub range the ultrasonic sensor sends out a pulse. It receives an echo from any solid object that is within a vicinity of about two meters. Which is just to be safe keeping distance. And we modulate that sound to make it sound a little clearer. Depending on the distance that the object is at, we can actually map the sound to different frequencies. So the closer the object is, the higher the pitch of the tone that's being played by the pick is. Cool. So that lets you know that something is much closer to you. This is kind of sort of inspired by the Doppler effect. Just because objects that are closer sound higher pitched. Sure. And the audio is played as three dimensional audio. So depending on the angle at which the obstacle is, the intensity between the ears is different as the time difference increases. So you're modeling, as the sound comes in, you're modeling the attenuation of that sound across the head and the delay in the sound reaching one ear versus the other ear. There are two components. There's the IIDA and the IVD. The intensity difference and time difference. And the intensity difference is the amplitude difference between the two ears. So the object on the right ear hears the sound louder than the left ear. And the time difference is that the right ear hears the sound slower than the left ear. Did you find that both were important? So for different angles and different frequencies, both of them were important. Interesting. Okay. I think the closer to the edges, the intensity matters more. And it also depends on the frequency. Like the formulas we found were taking the frequency that we need the sound to be as a parameter. And so it depends a little on that. And I think one thing was that in the beginning, before we had as much knowledge about the time delay. And that on its own was not as effective. Like you could see, I mean it wasn't really easy to determine. Okay. So the combination of both was... Interesting. Yeah. Cool. All right. So now for the demo, I guess. So we have this sort of demo environment. And what happens is that... Okay, sure. Right, right. If we were to put on headphones, the sound would be directional. But this will allow us just to hear it. Okay. That's very cool. So this one was the highest pitch. And then it decreased as you got further away from the objects. Very cool. It's interestingly reminiscent of like the stereotypical submarine thing or something. Yeah. So it sounds a little musical, right? Which makes it... Because I remember you guys experimenting with this and it was a little bit annoying. Yeah. Okay. Really cool. Could you run it one more time? And the other sort of interesting thing is... Uh-huh. Because it does sweep over like an angular range. Like if an object is closer, it obviously occupies a little bit more of the angular range. Like if you're more deep stuff, it... Sure. Okay. Awesome. Really cool. Thank you. If you want to try it, you guys can. Yeah. All right. More... So let's see. It doesn't matter which... So is this the correct or anything? Uh... Yeah. Right here. Okay. So that's the button here, which will start to saturate it. And so the way a user can know where to face in order... Yeah. The button is straight forward. I see. Okay. So like when you... Like if you're a visually impaired user, you would just place it such that the button is straight up... Okay. ...straight forward and that's how it would look. Okay. So I hold it facing straight forward and then push the button? Exactly. Yeah. The button has to face like exactly... I see. Okay. So that will probably be in standard range. Very cool. So it's... The noise sounds like what was coming out of the speaker, but it also has a directional element to it now. Yeah. So like each tone sounds like it's coming from... There was a tone from there, there, and there. Yeah. Very cool. Okay. Very cool. Now we have kind of a game. Okay. We're going to ask you to close your eyes... Okay. ...and rearrange the surroundings. Okay. And then... So my eyes are closed? Yes, and we're rearranging everything. And then you just move out of view because... Yeah, and then if you do the sweep facing straight forward... Uh-huh. ...with your eyes closed... Yeah. Can you tell us exactly where... What direction do you think the object is that you're adjusting? Let me run it one more time. Yeah. So I think there's... I think the closest object is here. Mm-hmm. Okay. And then the pitch is lowest here. Mm-hmm. So I think this must be the closest object. Right. Did I get it? Yes. Very cool. Awesome. Thank you, guys. We have another part of the game... ...where you get to arrange the surroundings... Okay. ...and then we can sort of... Let's see. Okay, that's you. All right. So... Close your eyes, Kritik. Yeah. All right. Can you start arranging? Mm-hmm. Okay. So I'll do it again. I think there's something very close here, right? Yeah, so in the center, there's something really close. And in the side, I think... Yeah, that's the farthest away, I think. And this is the second closest. Yeah. Yep. Oh, thank you. Awesome. Cool. Yeah, all right. Yeah, so, I mean, in the next, I guess... Like, do you have any questions for us? I mean, we were also thinking of describing some of the technical... Sure. Yeah, that would be great. I can take that if you want. Oh, okay. Yeah. Okay, so here... I mean, we already spoke about the sound. Yes. It is using DDS, like we did in Lab 1. With FM synthesis to make the... Yeah, okay. And then, again, the heavy-duty transfer function, we used the 3D audio with IRD and ITD. Uh-huh. And then, the distance-to-feature-to-mapping is done over three octaves, splitting the entire thing into six distinct pieces, and then mapping each one of them to... Right. Yeah, and the next segment was the distance. So, right now, we're using the ultrasonic sensor, which basically sends out a pulse and measures the echo and the duration of the echo is what determines how far the things are. And, like, we tried using some other sensors, like an IR and a DOF, but they were... Although it's more expensive, much more expensive, but they were less reliable. So, we ended up sticking with this because this was fairly accurate. And... What was the range that this product required? I was going to ask, yeah. Yeah. Like, yeah, we tested with different distances and to see what was measuring the best and this seemed to be the best option. Okay. So, we stuck with that. Yeah, and then, again, we used the server, like, a gimbal sort of structure to just move ourselves over the screen. So, we used, basically, WM... You can do it. ...inspired from Lab 3 to kind of control the distances that the server measures and, again, kind of used Lab 1's concept to, you know, actuate the entire system using a button because, yeah, that allows the user to control when they want to take their... Sure. ...observation instead of, like, having someone just continuously coming into that one all the time. And, yeah. A sort of critical thing that we thought about was just how, like, the different systems have, like, different timing constraints. So, you know, the ultrasonic sensor requires a far-screen sensor at a specific time interval. The servo requires a slower sort of PWM to move it over the range. And then, of course, the biggest timing constraint is ADS, which, you know, requires precise timing and needs cycles of time. So, we just kind of used, you know, different timers to synchronize the threads that are running each system. All the modules are... Like, the module concept of this is that they're all different threads, and they're all conditionally dependent on each other, so because we want things to execute in a certain order, like, we need to first measure the distance, then produce a sound according to that distance, and then move the servo. So all of these threads are conditionally dependent, and that's how they're synchronized. Yeah, and everything is driven by receiving input from the button, so that's also, like... I see. So the button starts the whole process, and then each process signals the subsequent process to... Exactly. Okay, okay. Another thing with the sensors is that the ultrasonic sensor goes crazy when there's something soft in front of it, so if humans... Oh, interesting. ...it really goes crazy. So you would be able to get a very high frequency of sound, and I think that's something that the LiDAR or EC1 can fix. Well, maybe... Could it be interpreted as a feature? Currently, we know that soft objects sound extremely high-pitched, so we know that there is a soft object at the specific location, but there's no way to tell the distance at which it's there, so it just sounds really high. So it's almost... Is it almost like a direction-to-person sensor? People tend to say that. But it's not necessarily a person. I mean, right now, we just kept all hard objects, but I think in some other places, we tested around the lab at some other objects when they let go. Sure, okay. Even a backpack would make it possible. Okay, okay, okay. Anything that... It's just the delay in the echo causes it to be a much higher frequency. I see, okay, okay. Yeah, and another thing was, just in terms of the form factor, we wanted it to be much more compact, and we're thinking about using the smaller PIC board, which you can just have on the stick, but because we're using the DAC and the DBS, we have to use this big one. And another thing is having the speakers because the headphones, most of the headphones, we tried with three different headphones and all of them didn't have enough gain to properly hear the sound, so that's why we have to use these speakers, but that could be fixed by having an explicit gain circuit that goes into the input of the headphones. Sure. Maybe the effect of the FM synthesis was prominent only when you use the speakers. When you use the speakers. Okay. Otherwise, don't sound very, very digital. Okay. Actually, all of this could be installed on a single board, but this is like most of the... It's more like a system in a box. Sure, yeah. But for sure, this could all be consolidated, but as a proof of concept, that's really compelling. Awesome. Okay. I don't think so. That was really a compelling demo. Thank you, guys.