OpenIPC
/
wiki
mirror of https://github.com/OpenIPC/wiki.git
1
0
Fork 0
Code Activity

Update fpv-audio.md

pull/359/head
Signor Pellegrino 2024-04-21 00:05:25 +03:00 committed by GitHub
parent 4a9bb59f65
commit 9b35745c4b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 10 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
en/fpv-audio.md
View File

@ -1,12 +1,12 @@
# Audio on FPV (Experimental) ## Audio on FPV (Experimental)
## Overview ### Overview
Everything has been tested using steamdeck (PC x86) and ssc338q-imx415 Anjoy board and choice of decoders and similar may reflect this. Everything has been tested using steamdeck (PC x86) and ssc338q-imx415 Anjoy board and choice of decoders and similar may reflect this.
Cúrrently there is ~200msec delay on audio stream, encoding settings may improve this (smaller packets, opus supports 20msec intervals), changing OS settings, changing to PCM/PCMA/PCMU. Cúrrently there is ~200msec delay on audio stream, encoding settings may improve this (smaller packets, opus supports 20msec intervals), changing OS settings, changing to PCM/PCMA/PCMU.
Most interesting observation is that the audio/video is perfectly synced in recorded TS file. It indicates delay comes from the decoder chain in GSTreamer. Most interesting observation is that the audio/video is perfectly synced in recorded TS file. It indicates delay comes from the decoder chain in GSTreamer.
If using mpegtsmux like I did, there is no need for decoding and parsing when using OPUS. If using mpegtsmux like I did, there is no need for decoding and parsing when using OPUS.
## Majestic and general settings ### Majestic and general settings
Audio settings (majestic.yaml): Audio settings (majestic.yaml):
``` ```
cli -s .audio.enabled true cli -s .audio.enabled true
@ -15,7 +15,7 @@ cli -s .audio.srate 8000 (goes up to 48000 but consumes more bandwidth)
Port can be set (default 5700) with: Port can be set (default 5700) with:
`cli -s .outgoing.audioPort 6200` (does it really work?) `cli -s .outgoing.audioPort 6200` (does it really work?)
## Quick wfb_ng setup ### Quick wfb_ng setup
Example wfb_ng setup: Example wfb_ng setup:
Drone: Drone:
``` ```
@ -26,28 +26,28 @@ GS:
wfb_rx -p 0 -c 127.0.0.1 -u 5700 -K /etc/gs.key -i 7669207 wlan1 wfb_rx -p 0 -c 127.0.0.1 -u 5700 -K /etc/gs.key -i 7669207 wlan1
``` ```
## Client side audio & video commands ### Client side audio & video commands
GST command for opus 8000hz (sound only): GST command for opus 8000hz (sound only):
``` ```
gst-launch-1.0 udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpopusdepay ! opusdec ! audioconvert ! autoaudiosink sync=false gst-launch-1.0 udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpopusdepay ! opusdec ! audioconvert ! autoaudiosink sync=false
``` ```
# Working sound, video & save!!!! ### Working sound, video & save!!!!
``` ```
gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtpjitterbuffer ! rtph265depay ! tee name=videoTee ! queue ! ts. udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpjitterbuffer ! rtpopusdepay ! tee name=audioTee ! queue ! ts. mpegtsmux name=ts ! filesink location=/run/media/deck/170a3e7f-325f-4567-8580-0e01dda76973/video/record-$(date +%y%m%d_%H%M%S).tsn sync=true -e videoTee. ! vaapih265dec ! fpsdisplaysink fps-update-interval=250 video-sink=autovideosink text-overlay=true sync=false audioTee. ! opusdec ! audioconvert ! pulsesink sync=false gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtpjitterbuffer ! rtph265depay ! tee name=videoTee ! queue ! ts. udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpjitterbuffer ! rtpopusdepay ! tee name=audioTee ! queue ! ts. mpegtsmux name=ts ! filesink location=/run/media/deck/170a3e7f-325f-4567-8580-0e01dda76973/video/record-$(date +%y%m%d_%H%M%S).tsn sync=true -e videoTee. ! vaapih265dec ! fpsdisplaysink fps-update-interval=250 video-sink=autovideosink text-overlay=true sync=false audioTee. ! opusdec ! audioconvert ! pulsesink sync=false
``` ```
## Video & sound play ### Video & sound play
``` ```
gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtph265depay ! vaapih265dec ! autovideosink sync=false udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpopusdepay ! opusdec ! audioconvert ! autoaudiosink sync=false gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtph265depay ! vaapih265dec ! autovideosink sync=false udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpopusdepay ! opusdec ! audioconvert ! autoaudiosink sync=false
``` ```
## Video & sound record ### Video & sound record
``` ```
gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtpjitterbuffer ! rtph265depay ! queue ! ts. udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpjitterbuffer ! rtpopusdepay ! queue ! ts. mpegtsmux name=ts ! filesink location=/run/media/deck/170a3e7f-325f-4567-8580-0e01dda76973/video/record-$(date +%y%m%d_%H%M%S).tsn sync=true -e gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265' ! rtpjitterbuffer ! rtph265depay ! queue ! ts. udpsrc port=5700 ! application/x-rtp, payload=98, encoding-name=OPUS ! rtpjitterbuffer ! rtpopusdepay ! queue ! ts. mpegtsmux name=ts ! filesink location=/run/media/deck/170a3e7f-325f-4567-8580-0e01dda76973/video/record-$(date +%y%m%d_%H%M%S).tsn sync=true -e
``` ```
FFPLAY command for opusn 8000hz (sound only): ### FFPLAY command for opusn 8000hz (sound only):
``` ```
ffplay.exe -fflags nobuffer -vn -flags low_delay -strict experimental -deadline realtime -framedrop -protocol_whitelist file,crypto,rtp,tcp,udp opus_audio.txt ffplay.exe -fflags nobuffer -vn -flags low_delay -strict experimental -deadline realtime -framedrop -protocol_whitelist file,crypto,rtp,tcp,udp opus_audio.txt
opus_audio.txt (sdp) opus_audio.txt (sdp)
@ -67,7 +67,7 @@ a=control:audio
a=rtpmap:98 opus/8000/1 a=rtpmap:98 opus/8000/1
``` ```
# TODO & suggestions: ### TODO & suggestions:
Optimize and adapt GST commands to different gs platforms. Optimize and adapt GST commands to different gs platforms.
### /etc/wfb.conf ### /etc/wfb.conf