From b4e2960e6711420ade3044a292ab0710fe3c6f38 Mon Sep 17 00:00:00 2001 From: snokvist <joakim.snokvist@gmail.com> Date: Sun, 5 May 2024 12:43:46 +0200 Subject: [PATCH] Update fpv-audio.md --- en/fpv-audio.md | 76 ++++++------------------------------------------- 1 file changed, 9 insertions(+), 67 deletions(-) diff --git a/en/fpv-audio.md b/en/fpv-audio.md index 2838caf..db7c866 100644 --- a/en/fpv-audio.md +++ b/en/fpv-audio.md @@ -1,79 +1,21 @@ -## Audio on FPV (Experimental) +## Audio on FPV ### Overview Everything has been tested using steamdeck (PC x86) and ssc338q-imx415 Anjoy board and choice of decoders and similar may reflect this. -Cúrrently there is ~200msec delay on audio stream, encoding settings may improve this (smaller packets, opus supports 20msec intervals), changing OS settings, changing to PCM/PCMA/PCMU. -Most interesting observation is that the audio/video is perfectly synced in recorded TS file. It indicates delay comes from the decoder chain in GSTreamer. -If using mpegtsmux like I did, there is no need for decoding and parsing when using OPUS. +Currently there is ~50-100ms latency, much imprroved from first trials ~200msec. The improvement comes mainy from using "pipewiresink" on client side, but it requires you to have a working pipewire audio backend on your GS. It could also be possible to use jack audio backend for similar or better result. +There was also an update to use a mixed RTP video/audio stream on the output port (standard 5600) which neccesitates a different approach for the gst pipeline (see below). The pipelines need to be disconnected in order to not block each other. +A small warning; if enabling audio and not adapting your pipeline to sort out payload=97 (H265) the RTP payload=98 will cause visual artifacts in your video. rtpjitterbuffer will help to manage out of order packets (will not work without it). +Video using OPUS 16000 OPUS samplerate: https://youtu.be/Z0KxSS24j7o ### Majestic and general settings Audio settings (majestic.yaml): ``` cli -s .audio.enabled true -cli -s .audio.srate 8000 (goes up to 48000 but consumes more bandwidth) -``` -Audio port can be set (default 5700) with: -``` -cli -s .outgoing.audioPort 6200 +cli -s .audio.srate 8000 (8000 pretty crap, 16000 usable and 48000 really good) ``` -### Quick wfb_ng setup -Example wfb_ng setup: -Drone: +### Working sound, video & save to file ``` -wfb_tx -p -u 5700 -K /etc/drone.key -k 8 -n 12 -i 7669207 -f data wlan0 +gst-launch-1.0 udpsrc port=5600 ! tee name=videoTee ! queue ! tee name=t t. ! queue ! application/x-rtp,payload=97, clock-rate=90000, encoding-name=H265 ! rtpjitterbuffer latency=20 ! rtph265depay ! mpegtsmux name=ts ! filesink location=/run/media/deck/170a3e7f-325f-4567-8580-0e01dda76973/video/record-$(date +%y%m%d_%H%M%S).tsn sync=false t. ! queue leaky=1 ! tee name=audioTee ! queue ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpjitterbuffer latency=22 do-lost=true drop-on-latency=true ! rtpopusdepay ! ts. audioTee. ! queue leaky=1 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpjitterbuffer latency=22 ! rtpopusdepay ! opusdec ! audioconvert ! audioresample ! pipewiresink blocksize=128 mode=render processing-deadline=0 sync=false async=false videoTee. ! queue ! application/x-rtp,payload=97, clock-rate=90000, encoding-name=H265 ! rtpjitterbuffer latency=20 ! rtph265depay ! vaapih265dec ! fpsdisplaysink fps-update-interval=200 video-sink=xvimagesink sync=false ``` -GS: -``` -wfb_rx -p 0 -c 127.0.0.1 -u 5700 -K /etc/gs.key -i 7669207 wlan1 -``` - -### Client side audio & video commands -GST command for opus 8000hz (sound only): -``` -gst-launch-1.0 udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpopusdepay ! opusdec ! audioconvert ! autoaudiosink sync=false -``` - -### Working sound, video & save -``` -gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtpjitterbuffer ! rtph265depay ! tee name=videoTee ! queue ! ts. udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpjitterbuffer ! rtpopusdepay ! tee name=audioTee ! queue ! ts. mpegtsmux name=ts ! filesink location=/run/media/deck/170a3e7f-325f-4567-8580-0e01dda76973/video/record-$(date +%y%m%d_%H%M%S).tsn sync=true -e videoTee. ! vaapih265dec ! fpsdisplaysink fps-update-interval=250 video-sink=autovideosink text-overlay=true sync=false audioTee. ! opusdec ! audioconvert ! pulsesink sync=false -``` - -### Video & sound play -``` -gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtph265depay ! vaapih265dec ! autovideosink sync=false udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpopusdepay ! opusdec ! audioconvert ! autoaudiosink sync=false -``` - -### Video & sound record -``` -gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtpjitterbuffer ! rtph265depay ! queue ! ts. udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpjitterbuffer ! rtpopusdepay ! queue ! ts. mpegtsmux name=ts ! filesink location=/run/media/deck/170a3e7f-325f-4567-8580-0e01dda76973/video/record-$(date +%y%m%d_%H%M%S).tsn sync=true -e -``` - -### FFPLAY command for opusn 8000hz (sound only): -``` -ffplay.exe -fflags nobuffer -vn -flags low_delay -strict experimental -deadline realtime -framedrop -protocol_whitelist file,crypto,rtp,tcp,udp opus_audio.txt -opus_audio.txt (sdp) -v=0 -o=StreamingServer 38990265062388 38990265062388 IN IP4 0.0.0.0 -s=RTSP Session -c=IN IP4 0.0.0.0 -t=0 0 -a=range:npt=0- -m=video 5600 RTP/AVP 97 -a=control:video -a=rtpmap:97 H265/90000 -a=fmtp:97 profile-level-id=016000; packetization-mode=1; -a=framerate:90 -m=audio 5700 RTP/AVP 98 -a=control:audio -a=rtpmap:98 opus/8000/1 -``` - -### TODO & suggestions: -Optimize and adapt GST commands to different gs platforms. - -### /etc/wfb.conf -add audio specific parameters like port, fec, ... - -### /usr/bin/wifibroadcast -add "if audio_enabled=true in datalink.conf then start wfb_tx ....." +Nothing more to do, it just works :-)