RTSP Client sometimes doesn't send video (stream 0) packets to the client

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

RTSP Client sometimes doesn't send video (stream 0) packets to the client

renjith.t
Hello,

We are using gstreamer rtsp server in our streaming server.

I would like to discuss & find a possible reason for one of the behaviors which we have observed while streaming our videos.

- Our Streaming Server is deployed in a Windows 10 PC
- We are using a single shared rtspmediafactory to stream our videos.
- a video streamed via the gstreamer rtsp server was displayed in different vlc clients in different machines (around 20-30 clients)
- we observed that in some clients the video was not displayed.

Our analysis

- We verified from wireshark(client end) that video packets were not received. audio was received
- We verified from wireshark(server end) that audio packets were sent to client, but video packets were not sent
- we observed that initially the video packets were sent/received for around 1 second but gstreamer stopped sending it (before stopping an ACK was received from the client with PSH bit on)
- On enabling the gst debugs a warning was printed "already a queued data message for channel 0" - from do_send_data_list (rtsp-client.c)
- though the stream in some clients were not displayed, in all other clients it was displayed properly


Our Troubleshooting techniques 

- we replaced the client with ODM ( same behavior was observed )
- we replaced the client PC ( same behavior was observed )
- we shifted to a private network ( still same behavior was observed in some clients) 


Configurations used
Camera Resolution :- 2048 X 1536 @ 30fps H.264 Codec
TCP stream was rendered to the clients
1000 Mbps switch was used
All client PC were windows 10 PC


We are stuck with this analysis and trying to find the exact reason for the video disconnection.

Please provide your valuable inputs for the above mentioned issue faced by us

Regards
Renjith Thankachan

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: RTSP Client sometimes doesn't send video (stream 0) packets to the client

Marc Leeman
Investigate the RTSP session management exchange first, maybe that
will contain valuable information.

e,g,

1. did the client request the video
2. what transport mechanism is used (I'm guessing RTP over TCP)
3. do you get the same behaviour when using UDP, what about UDP/muticast
...



On Wed, 10 Mar 2021 at 13:15, Renjith Thankachan (Software Development
- Telecom) <[hidden email]> wrote:

>
> Hello,
>
> We are using gstreamer rtsp server in our streaming server.
>
> I would like to discuss & find a possible reason for one of the behaviors which we have observed while streaming our videos.
>
> - Our Streaming Server is deployed in a Windows 10 PC
> - We are using a single shared rtspmediafactory to stream our videos.
> - a video streamed via the gstreamer rtsp server was displayed in different vlc clients in different machines (around 20-30 clients)
> - we observed that in some clients the video was not displayed.
>
> Our analysis
>
> - We verified from wireshark(client end) that video packets were not received. audio was received
> - We verified from wireshark(server end) that audio packets were sent to client, but video packets were not sent
> - we observed that initially the video packets were sent/received for around 1 second but gstreamer stopped sending it (before stopping an ACK was received from the client with PSH bit on)
> - On enabling the gst debugs a warning was printed "already a queued data message for channel 0" - from do_send_data_list (rtsp-client.c)
> - though the stream in some clients were not displayed, in all other clients it was displayed properly
>
>
> Our Troubleshooting techniques
>
> - we replaced the client with ODM ( same behavior was observed )
> - we replaced the client PC ( same behavior was observed )
> - we shifted to a private network ( still same behavior was observed in some clients)
>
>
> Configurations used
> Camera Resolution :- 2048 X 1536 @ 30fps H.264 Codec
> TCP stream was rendered to the clients
> 1000 Mbps switch was used
> All client PC were windows 10 PC
>
>
> We are stuck with this analysis and trying to find the exact reason for the video disconnection.
>
> Please provide your valuable inputs for the above mentioned issue faced by us
>
> Regards
> Renjith Thankachan
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel



--
g. Marc
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: RTSP Client sometimes doesn't send video (stream 0) packets to the client

renjith.t
Hi,

I have actually gone through the session media files.

What I found is that
-> gstreamer rtsp server is closing the transport for the stream 0
-> it is done from "update_transport" method in rtsp-stream.c
-> info debug printed is "removing TCP 192.168.111.78" where 111.78 was the
client where video was not displayed

so it is clear that gstreamer is explicitly closing the transport.





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

AW: RTSP Client sometimes doesn't send video (stream 0) packets to the client

Thornton, Keith
Hi,
that may be because the server is not receiving the keep alive timer from the client. If you have a wireshark log look to see if the client sends a GET_PARAMETER once a minute (If you haven't changed the keep alive timeout).
Gruesse

-----Ursprüngliche Nachricht-----
Von: gstreamer-devel <[hidden email]> Im Auftrag von renjith.t
Gesendet: Donnerstag, 11. März 2021 14:10
An: [hidden email]
Betreff: Re: RTSP Client sometimes doesn't send video (stream 0) packets to the client

Hi,

I have actually gone through the session media files.

What I found is that
-> gstreamer rtsp server is closing the transport for the stream 0 it is
-> done from "update_transport" method in rtsp-stream.c info debug
-> printed is "removing TCP 192.168.111.78" where 111.78 was the
client where video was not displayed

so it is clear that gstreamer is explicitly closing the transport.





--
Sent from: https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgstreamer-devel.966125.n4.nabble.com%2F&amp;data=04%7C01%7C%7C6e975d97adaa4a9d361508d8e4981c9b%7C28042244bb514cd680347776fa3703e8%7C1%7C1%7C637510689266574710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=jP9RcoouYXOZf1m1UAiUqAEuSn8TBvoy%2FpiGpFN%2BTkc%3D&amp;reserved=0
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fgstreamer-devel&amp;data=04%7C01%7C%7C6e975d97adaa4a9d361508d8e4981c9b%7C28042244bb514cd680347776fa3703e8%7C1%7C1%7C637510689266574710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=qscZFcyt5nNK836xWaQVbioQcpbQ9OyCAZPiGS1S6UQ%3D&amp;reserved=0
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: RTSP Client sometimes doesn't send video (stream 0) packets to the client

Jeff Shanab
I deal with a large amount of security cameras of different brands.

Just an FYI....
60 seconds and GET_PARAMETER are the default and work 90+% of the time.
The Setup reply will tell you if it is not default but is there for default also, depends on brand. end of session header "timeout=60"

I have seen cameras that
1) cannot handle GET_PARAMETER in any transport (RED Vision)
2) must have OPTIONS or SET_PARAMETER instead
3) use rtcp, the rest are ignored. (I thought the timeout=nn told me which but it is inconsistent)

The behaviour seems to vary with transport
1) rtsp-over-http.(2 sockets) All bets are off, every mfg interprets the vague combination of the Apple Quicktime Spec differently.
2) rtsp-over-tcp.(1 socket) Usually GET_PARAMETER is universally accepted here and is all that is needed.
3) rtsp/rtp/udp (up to 7 sockets) GET_PARAMETER for main session but some must have rtcp receiver reports per session or they disconnect,

Patterns I see are
   if disconnect in seconds, rtcp issue
   if right on the 30 second, 1 minutes or 2 minute then GET_PARAMETER/OPTIONS/SET_PARAMETER (OPTIONS on vary old cameras, otherwise OPTIONS tells you if it supports GET_PARAMETER

Wireshark is your friend.




On Thu, Mar 11, 2021 at 9:22 AM Thornton, Keith <[hidden email]> wrote:
Hi,
that may be because the server is not receiving the keep alive timer from the client. If you have a wireshark log look to see if the client sends a GET_PARAMETER once a minute (If you haven't changed the keep alive timeout).
Gruesse

-----Ursprüngliche Nachricht-----
Von: gstreamer-devel <[hidden email]> Im Auftrag von renjith.t
Gesendet: Donnerstag, 11. März 2021 14:10
An: [hidden email]
Betreff: Re: RTSP Client sometimes doesn't send video (stream 0) packets to the client

Hi,

I have actually gone through the session media files.

What I found is that
-> gstreamer rtsp server is closing the transport for the stream 0 it is
-> done from "update_transport" method in rtsp-stream.c info debug
-> printed is "removing TCP 192.168.111.78" where 111.78 was the
client where video was not displayed

so it is clear that gstreamer is explicitly closing the transport.





--
Sent from: https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgstreamer-devel.966125.n4.nabble.com%2F&amp;data=04%7C01%7C%7C6e975d97adaa4a9d361508d8e4981c9b%7C28042244bb514cd680347776fa3703e8%7C1%7C1%7C637510689266574710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=jP9RcoouYXOZf1m1UAiUqAEuSn8TBvoy%2FpiGpFN%2BTkc%3D&amp;reserved=0
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fgstreamer-devel&amp;data=04%7C01%7C%7C6e975d97adaa4a9d361508d8e4981c9b%7C28042244bb514cd680347776fa3703e8%7C1%7C1%7C637510689266574710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=qscZFcyt5nNK836xWaQVbioQcpbQ9OyCAZPiGS1S6UQ%3D&amp;reserved=0
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: RTSP Client sometimes doesn't send video (stream 0) packets to the client

Marc Leeman
Unless 'OPTIONS' lists commands that it does not support :-)

The joy of network cameras...

On Thu, 11 Mar 2021 at 16:45, Jeff Shanab <[hidden email]> wrote:

>
> I deal with a large amount of security cameras of different brands.
>
> Just an FYI....
> 60 seconds and GET_PARAMETER are the default and work 90+% of the time.
> The Setup reply will tell you if it is not default but is there for default also, depends on brand. end of session header "timeout=60"
>
> I have seen cameras that
> 1) cannot handle GET_PARAMETER in any transport (RED Vision)
> 2) must have OPTIONS or SET_PARAMETER instead
> 3) use rtcp, the rest are ignored. (I thought the timeout=nn told me which but it is inconsistent)
>
> The behaviour seems to vary with transport
> 1) rtsp-over-http.(2 sockets) All bets are off, every mfg interprets the vague combination of the Apple Quicktime Spec differently.
> 2) rtsp-over-tcp.(1 socket) Usually GET_PARAMETER is universally accepted here and is all that is needed.
> 3) rtsp/rtp/udp (up to 7 sockets) GET_PARAMETER for main session but some must have rtcp receiver reports per session or they disconnect,
>
> Patterns I see are
>    if disconnect in seconds, rtcp issue
>    if right on the 30 second, 1 minutes or 2 minute then GET_PARAMETER/OPTIONS/SET_PARAMETER (OPTIONS on vary old cameras, otherwise OPTIONS tells you if it supports GET_PARAMETER
>
> Wireshark is your friend.
>
>
>
>
> On Thu, Mar 11, 2021 at 9:22 AM Thornton, Keith <[hidden email]> wrote:
>>
>> Hi,
>> that may be because the server is not receiving the keep alive timer from the client. If you have a wireshark log look to see if the client sends a GET_PARAMETER once a minute (If you haven't changed the keep alive timeout).
>> Gruesse
>>
>> -----Ursprüngliche Nachricht-----
>> Von: gstreamer-devel <[hidden email]> Im Auftrag von renjith.t
>> Gesendet: Donnerstag, 11. März 2021 14:10
>> An: [hidden email]
>> Betreff: Re: RTSP Client sometimes doesn't send video (stream 0) packets to the client
>>
>> Hi,
>>
>> I have actually gone through the session media files.
>>
>> What I found is that
>> -> gstreamer rtsp server is closing the transport for the stream 0 it is
>> -> done from "update_transport" method in rtsp-stream.c info debug
>> -> printed is "removing TCP 192.168.111.78" where 111.78 was the
>> client where video was not displayed
>>
>> so it is clear that gstreamer is explicitly closing the transport.
>>
>>
>>
>>
>>
>> --
>> Sent from: https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgstreamer-devel.966125.n4.nabble.com%2F&amp;data=04%7C01%7C%7C6e975d97adaa4a9d361508d8e4981c9b%7C28042244bb514cd680347776fa3703e8%7C1%7C1%7C637510689266574710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=jP9RcoouYXOZf1m1UAiUqAEuSn8TBvoy%2FpiGpFN%2BTkc%3D&amp;reserved=0
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fgstreamer-devel&amp;data=04%7C01%7C%7C6e975d97adaa4a9d361508d8e4981c9b%7C28042244bb514cd680347776fa3703e8%7C1%7C1%7C637510689266574710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=qscZFcyt5nNK836xWaQVbioQcpbQ9OyCAZPiGS1S6UQ%3D&amp;reserved=0
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel



--
g. Marc
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: RTSP Client sometimes doesn't send video (stream 0) packets to the client

renjith.t
Hello All,

Its not an issue with the GET_PARAMETERS Request.
GET_PARAMETERS request is sent by the client, it is also received and
processed further by the rtsp server.

We took time to analyze this issue, and below is our observation

- gst_rtsp_watch_write_serialized_messages :- function writes the stream on
to the socket.
- If the messages to be sent to the clients is in big numbers, then some
messages are not sent to the client, this function adds the pending messages
to the client's watch and allocates the  dispatcher
(gst_rtsp_source_dispatch_write) for sending those pending messages, and
sets a sequence no in the last message.
- when gst_rtsp_source_dispatch_write is invoked, it is responsible to send
those pending messages to the client. when all messages are sent
successfully, the callback for message_sent is called, which in-turn will
reset the sequence number.


*Now the problem arises in the below situation* (it occurs when the number
of TCP clients for a single shared stream is huge - I reproduced this issue
with 10-15 TCP Clients (5-7 clients per PC) with resolution of 2048 X 1536 @
25fps H.264 Codec )

- gst_rtsp_watch_write_serialized_messages  was not able to write the
messages completely
- It added the pending messages to the client's watch & assigned the
dispatcher for it.
- the above was done for all the 5-6 clients in the same pc
- now the dispatcher came in into action
- dispatcher took clients one by one and started to send the messages from
the client's watch
- before the dispatcher could start sending messages to the last client,
next message for that client was scheduled to be sent through
gst_rtsp_watch_write_serialized_messages (*I believe it is a case of context
switching* )
- now send_tcp_message of rtsp-stream.c came into action.
- it tried to send message to the client, from check_transport_backlog where
it failed
- on failing it removed the transport of the client. using update_transport
- hence no video was displayed on the client side since then.
- after that the dispatcher came into picture once again and delivered the
message to the client
- it was only after the transport was removed, the client's object received
the message sent notification for the sequence which was set earlier

On a trial basis, i stopped the removing the transport, it worked fine with
15-20 clients for around 15 hrs

Other observation i came across was
- every time the dispatcher was called from the same thread (even for
different clients of different streams)
- the same thread was also used to send messages to the other clients also.


Any help or solution to this problem is heartily welcomed.
Best Regards



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel