Microphone / audio over SFS with Unity 3.5

Post here your questions about the Unity / .Net / Mono / Windows 8 / Windows Phone 8 API for SFS2X

Moderators: Lapo, Bax

ThomasLund
Posts: 1297
Joined: 14 Mar 2008, 07:52
Location: Sweden

Microphone / audio over SFS with Unity 3.5

Postby ThomasLund » 21 Apr 2012, 08:43

Hey hey,

For a customer project I'm starting up a small networked audio script taking Microphone from 3.5 and sending it over SFS server to all clients in the same room.

And I wondered if anyone had done something like this before, so that it might save me (and others) some work. I'd be open to share my findings here for everyone - if you are too :-D End of the day we could have a nice small example code snippet.

I'll start out, and lets see if someone jumps on board.

The task
1) On recording client side
Start recording audio from Unity Microphone
As its being recorded, grab data from the audio clip in chunks
"Do something with chunk" to compress it for network transport, maybe filtering out noise, resample etc
Send it as an object message package to SFS server for distribution

2) On receiving/playing client side
Assemble audio data into some playable format for Unity audio clip
Play it continuously as there is data arriving
If there are multiple senders, need to play multiple samples (3d positioned? merged together?)

The challenges
* Recording and playing audio
Very simple. http://unity3d.com/support/documentatio ... phone.html

We get it as an audio clip and can simply read its content. Assemble it as an audio clip on the other end and play it on an audio source. Easy.

* Quality and compression
In a LAN environment we can simply send the audio I think raw. But once you start to send over the internet, there is going to be bandwidth issues. So need to find a way to compress the audio data.

We can also try to not sample at high rate, but just take some lower sampling rate. For voice I bet its possible to do it much lower than music and sfx.

That leaves the big one on compression - I have never written and audio compressor in C# (or any other language). And it also has to be something we can decompress into an audio clip supported data format.

* Cross platform
Since Unity's Microphone works for all devices except Android (if I'm not mistaken), it should be very simple to do this cross platform with the same code. But we cannot e.g. use speex or similar, as thats external plugin that wont work in a webplayer. Would be cool to have, but wont happen unless there is a 100% managed code version out there.

On Android - too bad. Need to wait for Unity to support it

* Network transport
The SFS room setup can be used here together with users being able to be inside multiple rooms at the same time. So we can have a given user in the "game room" whatever that is, where the regular game code is run. We can then have users being joined to that particulars "game microphone audio exchange" room. Clients connected to that will receive all spoken audio. If the client doesnt want that - leave that room and he wont be bothered. So very simple and effective.

As we then simply send the data with object messaging to this "game microphone audio exchange" room, everyone will get the data that needs it.

* Other things I've forgotten?

Some links
Microphone: http://unity3d.com/support/documentatio ... phone.html
AudioClip: http://unity3d.com/support/documentatio ... oClip.html
Someone tinkering with it for Unity networking: http://forum.unity3d.com/threads/123776 ... twork-Test
Maybe relevant (or not) - saving as wav file: http://forum.unity3d.com/threads/119295 ... av-problem
Sending SFS object messages: http://docs2x.smartfoxserver.com/api-do ... 36afe0.htm

Getting started
Anyone interested? Both in using and contributing? Raise your hand :-D

If no one, then I'll just make this myself and not share it. But doesnt help the next person

But basically I think one should start off the Microphone Network Test snippet from the Unity forums that I linked above. It doesnt address a lot of the issues, but it could be used as basis for raw audio over LAN net. Then work from there.

/Thomas
Full Control - maker of Unity/C# and Java SFS API and indie games
Follow on twitter: http://twitter.com/thomas_h_lund
appels
Posts: 464
Joined: 28 Jul 2010, 02:12
Contact:

Re: Microphone / audio over SFS with Unity 3.5

Postby appels » 24 Apr 2012, 05:58

Great initiative, i had a go at it last weekend but didn't get very far but then again I didn't have much time either.
I will try again in the next few days.
I started of with the example from the forum.
I'll update when i have new info ...
ThomasLund
Posts: 1297
Joined: 14 Mar 2008, 07:52
Location: Sweden

Re: Microphone / audio over SFS with Unity 3.5

Postby ThomasLund » 06 Jul 2012, 13:31

Got this working with uLaw compression, server extension and iOS/webplayer/standalone microphone support.

Super nice - still have to run some tests and measuring performance.

/T
Full Control - maker of Unity/C# and Java SFS API and indie games

Follow on twitter: http://twitter.com/thomas_h_lund
appels
Posts: 464
Joined: 28 Jul 2010, 02:12
Contact:

Re: Microphone / audio over SFS with Unity 3.5

Postby appels » 18 Jul 2012, 05:37

Hi Thomas, I will test it out tonight and let you know.
ThomasLund
Posts: 1297
Joined: 14 Mar 2008, 07:52
Location: Sweden

Re: Microphone / audio over SFS with Unity 3.5

Postby ThomasLund » 19 Jul 2012, 08:34

Extension is in your email :-)
Full Control - maker of Unity/C# and Java SFS API and indie games

Follow on twitter: http://twitter.com/thomas_h_lund
isil
Posts: 1
Joined: 19 Jul 2012, 13:34

Re: Microphone / audio over SFS with Unity 3.5

Postby isil » 19 Jul 2012, 13:36

Hi, if you need an other person for test let me know could be very interesting.
Good work :)

Return to “SFS2X C# API”

Who is online

Users browsing this forum: No registered users and 4 guests