RTP+MJPEG - Sniffing packets with wireshark, How clients join the rtp server

[Mauro Brenna]

Hello,

I am studying the RTP protocol in order to implement a server for RTP
with MJPEG payload using RFC 2435.
I read the RFC and I tried to stream and receive a video in rtp using gstreamer.

I read about :
http://www.gstreamer.net/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-rtpjpegpay.html
which is exactly the implementation of the RFC I read.

I tested the streaming and reception of two camera outputs using a
pipeline which I have found in the Internet and which seems to work:

#Streaming rtp mjpeg
gst-launch-0.10 -v gstrtpbin name=rtpbin v4l2src device=/dev/video0 !
video/x-raw-rgb,width=640,height=480,framerate=\(fraction\)30/1 !
ffmpegcolorspace ! jpegenc ! rtpjpegpay ! rtpbin.send_rtp_sink_0
rtpbin.send_rtp_src_0 ! multiudpsink clients="127.0.0.1:9996"
rtpbin.send_rtcp_src_0 ! multiudpsink clients="127.0.0.1:9997"
sync=false async=false udpsrc port=10000 ! rtpbin.recv_rtcp_sink_0
v4l2src device=/dev/video1 !
video/x-raw-rgb,width=640,height=480,framerate=\(fraction\)30/1 !
ffmpegcolorspace ! jpegenc ! rtpjpegpay ! rtpbin.send_rtp_sink_1
rtpbin.send_rtp_src_1 ! multiudpsink clients="127.0.0.1:9998"
rtpbin.send_rtcp_src_1 ! multiudpsink clients="127.0.0.1:9999"
sync=false async=false udpsrc port=10001 ! rtpbin.recv_rtcp_sink_1

#Playing rtp mjpeg
gst-launch-0.10 -v gstrtpbin name=rtpbin udpsrc
caps="application/x-rtp, media=\(string\)video,
clock-rate=\(int\)90000, encoding-name=\(string\)JPEG,
ssrc=\(guint\)469657143, payload=\(int\)96,
clock-base=\(guint\)2841649723, seqnum-base=\(guint\)39869" port=9996
! rtpbin.recv_rtp_sink_1  rtpbin. ! rtpjpegdepay ! jpegdec ! videorate
! video/x-raw-yuv ,framerate=30/1 !  ffmpegcolorspace  ! xvimagesink
udpsrc port=9997 ! rtpbin.recv_rtcp_sink_1 rtpbin.send_rtcp_src_1 !
multiudpsink clients="127.0.0.1:10001" sync=false async=false &
gst-launch-0.10 -v gstrtpbin name=rtpbin udpsrc
caps="application/x-rtp, media=\(string\)video,
clock-rate=\(int\)90000, encoding-name=\(string\)JPEG,
ssrc=\(guint\)469657143, payload=\(int\)96,
clock-base=\(guint\)2841649723, seqnum-base=\(guint\)39869" port=9998
! rtpbin.recv_rtp_sink_1  rtpbin. ! rtpjpegdepay ! jpegdec ! videorate
! video/x-raw-yuv ,framerate=30/1 !  ffmpegcolorspace  ! xvimagesink
udpsrc port=9999 ! rtpbin.recv_rtcp_sink_1 rtpbin.send_rtcp_src_1 !
multiudpsink clients="127.0.0.1:10001" sync=false async=false


Everything works but I would like to ask to this mailinglist a couple
of questions about the implementation and RTP itself:

1) I tried to sniff the packets using wireshark. I can see the packets
that are sent using UDP but not the RTP encapsulation (I see only the
field data of the UDP packet). Is is a problem of wireshark for RTP
(RFC 2435) or the gstreamer plug-in is not compliant to the standard?
I tested the same thing in VLC that streams RTP packets using RFC
2250. In wireshark I can see the RTP packet with the correct payload
type.

2) I do not understand very well how the client (player) join the RTP
server (streamer). I read the general RFC 1889. In the section about
RTP Control Protocol, it is not clear how the initialization works. I
mean, when I start the streamer, does it start to send RTP+MJPEG
packets to the unicast/multicast destination even if nobody is
listening to it? Or does it start sending packets after the reception
of a particular packet from the client ? I read about feedbacks
packets from sender/receiver and about the BYE (Goodbye RTCP packet).
However, I have not found a sort of JOIN packet.


Thanks in advance for the replies,

Mauro

------------------------------------------------------------------------------
Lotusphere 2011
Register now for Lotusphere 2011 and learn how
to connect the dots, take your collaborative environment
to the next level, and enter the era of Social Business.
http://p.sf.net/sfu/lotusphere-d2d
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Marco Ballesio]

Hi,

these are really generic RTP/wireshark specific questions so I might
not be the more appropriate guy on this planet to reply, btw a few
answers below..

On Tue, Dec 21, 2010 at 12:16 PM, Mauro Brenna <[hidden email]> wrote:

..snip..

>
>
> Everything works but I would like to ask to this mailinglist a couple
> of questions about the implementation and RTP itself:
>
> 1) I tried to sniff the packets using wireshark. I can see the packets
> that are sent using UDP but not the RTP encapsulation (I see only the
> field data of the UDP packet). Is is a problem of wireshark for RTP
> (RFC 2435) or the gstreamer plug-in is not compliant to the standard?
> I tested the same thing in VLC that streams RTP packets using RFC
> 2250. In wireshark I can see the RTP packet with the correct payload
> type.

It's usually up to a sender/receiver negotiation to agree that an UDP
communication is actually RTP and which are its parameters. Standard
ways to perform such a negotiation are RTSP, SIP/SDP, XMPP, etc. etc.
. Afaik, by no means this negotiation, aka "the session initiation" or
"signaling phase", is described in RFC1889.

As such a negotiation is not mandatory, in case the two ends of the
communication "already know" the basic parameters of the stream
(basically, destination IP and port) the latter can happily occur
without any violation to the standards.

Now, what has Wireshark to do with all of this? Basically, it is a
very smart tool which, other than sniffing the network traffic, also
processes it (maybe you noted it in the lower part of its UI ;) ).
This happens also for the negotiation packets and so, when it
discovers that an UDP stream matches with the parameters of, for
instance, a previous RTSP session initiation (appearing in the network
dump), it's "dissected" as RTP.

This kind of dissection fails in a few cases:

1) No negotiation occurred, as in the pipelines you posted above.
2) The negotiation protocol in unknown to Wireshark. For instance, you
designed and implemented it by yourself: in such a case you'll have to
pay the fee for doing proprietary stuff :).
3) The capture Wireshark is analysing began AFTER a "dissectable"
negotiation, so the tool does not have a clue about its contents.

>
> 2) I do not understand very well how the client (player) join the RTP
> server (streamer). I read the general RFC 1889. In the section about
> RTP Control Protocol, it is not clear how the initialization works.

RTCP is not about session initiation, as RFC1889 says, it is "to allow
monitoring of the data delivery".  Read above for more details.

> I
> mean, when I start the streamer, does it start to send RTP+MJPEG
> packets to the unicast/multicast destination even if nobody is
> listening to it?

Yes, it does. How could it happen without signaling otherwise?

> Or does it start sending packets after the reception
> of a particular packet from the client ? I read about feedbacks
> packets from sender/receiver and about the BYE (Goodbye RTCP packet).
> However, I have not found a sort of JOIN packet.
>

Because -unless you're using means other than RFC1889- there aren't
any. Probably VLC performs a session initiation (I guess RTSP, you
should see it in the network capture). If you need such a feature, on
the GStreamer side you'll need to use the "rtspsrc" element.

Regards

>
> Thanks in advance for the replies,
>
> Mauro
>
> ------------------------------------------------------------------------------
> Lotusphere 2011
> Register now for Lotusphere 2011 and learn how
> to connect the dots, take your collaborative environment
> to the next level, and enter the era of Social Business.
> http://p.sf.net/sfu/lotusphere-d2d
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

------------------------------------------------------------------------------
Forrester recently released a report on the Return on Investment (ROI) of
Google Apps. They found a 300% ROI, 38%-56% cost savings, and break-even
within 7 months.  Over 3 million businesses have gone Google with Google Apps:
an online email calendar, and document program that's accessible from your
browser. Read the Forrester report: http://p.sf.net/sfu/googleapps-sfnew
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Marco Ballesio]

Hi,

one note I forgot to write before (sorry for the spam)..

On Wed, Dec 22, 2010 at 10:03 AM, Marco Ballesio <[hidden email]> wrote:

..snip..

>> 2) I do not understand very well how the client (player) join the RTP
>> server (streamer). I read the general RFC 1889. In the section about
>> RTP Control Protocol, it is not clear how the initialization works.
>
> RTCP is not about session initiation, as RFC1889 says, it is "to allow
> monitoring of the data delivery".  Read above for more details.

And also, unless you're dealing with very old HW/SW, you should use
RFC 3550 in place of RFC 1889 (not that changes are so drastic
though).

Regards

>
>> I
>> mean, when I start the streamer, does it start to send RTP+MJPEG
>> packets to the unicast/multicast destination even if nobody is
>> listening to it?
>
> Yes, it does. How could it happen without signaling otherwise?
>
>> Or does it start sending packets after the reception
>> of a particular packet from the client ? I read about feedbacks
>> packets from sender/receiver and about the BYE (Goodbye RTCP packet).
>> However, I have not found a sort of JOIN packet.
>>
>
> Because -unless you're using means other than RFC1889- there aren't
> any. Probably VLC performs a session initiation (I guess RTSP, you
> should see it in the network capture). If you need such a feature, on
> the GStreamer side you'll need to use the "rtspsrc" element.
>
> Regards
>
>>
>> Thanks in advance for the replies,
>>
>> Mauro
>>
>> ------------------------------------------------------------------------------
>> Lotusphere 2011
>> Register now for Lotusphere 2011 and learn how
>> to connect the dots, take your collaborative environment
>> to the next level, and enter the era of Social Business.
>> http://p.sf.net/sfu/lotusphere-d2d
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>

------------------------------------------------------------------------------
Forrester recently released a report on the Return on Investment (ROI) of
Google Apps. They found a 300% ROI, 38%-56% cost savings, and break-even
within 7 months.  Over 3 million businesses have gone Google with Google Apps:
an online email calendar, and document program that's accessible from your
browser. Read the Forrester report: http://p.sf.net/sfu/googleapps-sfnew
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Mauro Brenna]

Hi,

thank you for the replies.

I looked for some more information about wireshark and it appears that
wireshark can detect the RFC 2435 (RTP for MJPEG).

http://www.wireshark.org/docs/dfref/j/jpeg.html

So, I expected, like it is for VLC using RFC 2250, that every UDP
packet containing an RTP packet as payload will be detected. During my
tests, I see only standard UDP packets.

I read something more about RTP and JPEG, I would like to know if I
can use the RFC 2435 for images and another RFC for audio and
synchronize audio and video using RTCP. In this case, I would like to
know if I can do the same for N video sources in RFC 2435. In the
negative case I believe I will be constraint to use RFC 2250 for
streaming both video and audio.
Moreover, can I stream directly JPEG frames using RFC2250? IMHO,Since
it isn't written in the RFC I believe it isn't possible.

I know that these questions are a bit OT because they are more general
than gstreamer use, but I am trying to understand the real
potentiality of the protocol for developing my realtime application.


Thanks,

Mauro

------------------------------------------------------------------------------
Forrester recently released a report on the Return on Investment (ROI) of
Google Apps. They found a 300% ROI, 38%-56% cost savings, and break-even
within 7 months.  Over 3 million businesses have gone Google with Google Apps:
an online email calendar, and document program that's accessible from your
browser. Read the Forrester report: http://p.sf.net/sfu/googleapps-sfnew
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Marco Ballesio]

Hi,

On Wed, Dec 22, 2010 at 1:20 PM, Mauro Brenna <[hidden email]> wrote:
> Hi,
>
> thank you for the replies.
>
> I looked for some more information about wireshark and it appears that
> wireshark can detect the RFC 2435 (RTP for MJPEG).
>
> http://www.wireshark.org/docs/dfref/j/jpeg.html

"detect" is the wrong term, at least in the context of the link you
sent. Wireshark can "display" the content of these packets or, that is
the same, it has a dissector for the protocol.

In my (limited) experience with the tool, the automagic detection
occurs only if you have a signalling phase Wireshark understands.

You want to just "see" the contents and you don't care about
automagic, right-click a packet containing RFC2435 -> "Decode as.." ->
"RTP" in the scroll down list. For more questions about Wireshark,
please refer to the appropriate contacts:

http://www.wireshark.org/lists/

>
> So, I expected, like it is for VLC using RFC 2250, that every UDP
> packet containing an RTP packet as payload will be detected. During my
> tests, I see only standard UDP packets.

Please, check if you have a signalling phase in the VLC capture. It's
very easy, you should see some "RTSP" packets going back and forth
before the actual stream begins.

>
> I read something more about RTP and JPEG, I would like to know if I
> can use the RFC 2435 for images and another RFC for audio

git clone git://anongit.freedesktop.org/gstreamer/gst-plugins-good
grep -r RFC gst-plugins-good/gst/rtp | grep audio

gives me:

gst-plugins-good/gst/rtp/gstrtpL16pay.c:      "Payload-encode Raw
audio into RTP packets (RFC 3551)",
gst-plugins-good/gst/rtp/gstrtpac3depay.c:      "Extracts AC3 audio
from RTP packets (RFC 4184)",
gst-plugins-good/gst/rtp/gstrtpamrdepay.c:      "Extracts AMR or
AMR-WB audio from RTP packets (RFC 3267)",
gst-plugins-good/gst/rtp/gstrtpamrpay.c:      "Payload-encode AMR or
AMR-WB audio into RTP packets (RFC 3267)",
gst-plugins-good/gst/rtp/gstrtpbvdepay.c:      "Extracts BroadcomVoice
audio from RTP packets (RFC 4298)",
gst-plugins-good/gst/rtp/gstrtpbvpay.c:      "Packetize BroadcomVoice
audio streams into RTP packets (RFC 4298)",
gst-plugins-good/gst/rtp/gstrtpg722pay.c:      "Payload-encode Raw
audio into RTP packets (RFC 3551)",
gst-plugins-good/gst/rtp/gstrtpg723depay.c:      "Extracts G.723 audio
from RTP packets (RFC 3551)",
gst-plugins-good/gst/rtp/gstrtpg723pay.c:    GST_STATIC_CAPS
("audio/G723, "     /* according to RFC 3551 */
gst-plugins-good/gst/rtp/gstrtpg729depay.c:      "Extracts G.729 audio
from RTP packets (RFC 3551)",
gst-plugins-good/gst/rtp/gstrtpg729pay.c:    GST_STATIC_CAPS
("audio/G729, "     /* according to RFC 3555 */
gst-plugins-good/gst/rtp/gstrtpilbcdepay.c:      "Extracts iLBC audio
from RTP packets (RFC 3952)",
gst-plugins-good/gst/rtp/gstrtpmp4adepay.c:      "Extracts MPEG4 audio
from RTP packets (RFC 3016)",
gst-plugins-good/gst/rtp/gstrtpmp4apay.c:      "Payload MPEG4 audio as
RTP packets (RFC 3016)",
gst-plugins-good/gst/rtp/gstrtpmpadepay.c:      "Extracts MPEG audio
from RTP packets (RFC 2038)",
gst-plugins-good/gst/rtp/gstrtpmpapay.c:      "Payload MPEG audio as
RTP packets (RFC 2038)",
gst-plugins-good/gst/rtp/gstrtpmparobustdepay.c:      "Extracts MPEG
audio from RTP packets (RFC 5219)",
gst-plugins-good/gst/rtp/gstrtpqcelpdepay.c:      "Extracts QCELP
(PureVoice) audio from RTP packets (RFC 2658)",
gst-plugins-good/gst/rtp/gstrtpqdmdepay.c:      "Extracts QDM2 audio
from RTP packets (no RFC)",
gst-plugins-good/gst/rtp/gstrtpvorbispay.c:      "Payload-encode
Vorbis audio into RTP packets (RFC 5215)",

(Disclaimer: that is not a comprehensive list)

> and
> synchronize audio and video using RTCP.

RTP is actually designed to grant synchronisation. Unless you've some
peculiar requirements, all you have to use is a proper GStreamer
pipeline with gstrtpbin.

> In this case, I would like to
> know if I can do the same for N video sources in RFC 2435.

In theory, yes. In practice, I never tried to synchronise multiple
video streams over RTP (and you mat run into issues like synchronising
the clock of the various source devices).

> In the
> negative case I believe I will be constraint to use RFC 2250 for
> streaming both video and audio.

RFC 2250 says "RTP Payload Format for MPEG1/MPEG2 Video". Taking apart
the fact it is only for video and quite obsolete, are you limited to
this technology because of customer requirements? You could achieve
better quality (and lower bandwidth) using, for instance, h264 and
rfc3984.

> Moreover, can I stream directly JPEG frames using RFC2250? IMHO,Since
> it isn't written in the RFC I believe it isn't possible.

on the same folder of before,

grep -r RFC gst-plugins-good/gst/rtp | grep JPEG

gives some answers ;).

>
> I know that these questions are a bit OT because they are more general
> than gstreamer use, but I am trying to understand the real
> potentiality of the protocol for developing my realtime application.

Sure it depends on how "standard" are your requirements. I've seen
nothing too complex from your descriptions so far.

Regards

>
>
> Thanks,
>
> Mauro
>
> ------------------------------------------------------------------------------
> Forrester recently released a report on the Return on Investment (ROI) of
> Google Apps. They found a 300% ROI, 38%-56% cost savings, and break-even
> within 7 months.  Over 3 million businesses have gone Google with Google Apps:
> an online email calendar, and document program that's accessible from your
> browser. Read the Forrester report: http://p.sf.net/sfu/googleapps-sfnew
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

------------------------------------------------------------------------------
Forrester recently released a report on the Return on Investment (ROI) of
Google Apps. They found a 300% ROI, 38%-56% cost savings, and break-even
within 7 months.  Over 3 million businesses have gone Google with Google Apps:
an online email calendar, and document program that's accessible from your
browser. Read the Forrester report: http://p.sf.net/sfu/googleapps-sfnew
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel