Project Home
Project Home
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - virtual circuit close timeout: (12 Items)
   
virtual circuit close timeout  
Hi,

We have encountered a problem with timeout when a virtual circuit is being closed.

There are 2 machines connected using FLEET.

On one machine there was a hardware problem and Fsys.atapi crashed.

At some later time on the second machine 'sin ne' is executed.
Process sin becomes indefinitely blocked on virtual circuit to Fsys on other machine.
Then when interrupt signal (ctrl+c) is sent to the sin process, the Proc is closing the virtual circuit with 1 minute 
timeout.

During this timeout the second machine is unable to create or terminate processes which is the real problem.

We are able to reproduce the problem by stopping Fsys driver on the machine using 'slay -sstop Fsys.atapi'.
One way to induce the create/terminate process timeout is the 'sin ne' but there are other commands which can induce it.


Bottom line is that a problem on one machine can introduce a "bigger" problem on another machine.
Any advice?

Thanks,
stepan hejny
Re: virtual circuit close timeout  
Hi Stepan,

I can’t reproduce your issue. I’m using two virtual PC which are connected by FLEET. At one machine I execute slay -s 
stop Fsys.atapi, then on second machine I run sin ne. I don’t see timeout.

Can you send me the following output?

sin ver
sin ar
sin ir
sin in
show_pci -vvv

Do you reproduce this issue on specific hardware or on different configurations?

Respectfully,
Oleg

19 янв. 2017 г., в 15:17, s h <community-noreply@qnx.com> написал:

> Hi,
> 
> We have encountered a problem with timeout when a virtual circuit is being closed.
> 
> There are 2 machines connected using FLEET.
> 
> On one machine there was a hardware problem and Fsys.atapi crashed.
> 
> At some later time on the second machine 'sin ne' is executed.
> Process sin becomes indefinitely blocked on virtual circuit to Fsys on other machine.
> Then when interrupt signal (ctrl+c) is sent to the sin process, the Proc is closing the virtual circuit with 1 minute 
timeout.
> 
> During this timeout the second machine is unable to create or terminate processes which is the real problem.
> 
> We are able to reproduce the problem by stopping Fsys driver on the machine using 'slay -sstop Fsys.atapi'.
> One way to induce the create/terminate process timeout is the 'sin ne' but there are other commands which can induce 
it.
> 
> Bottom line is that a problem on one machine can introduce a "bigger" problem on another machine.
> Any advice?
> 
> Thanks,
> stepan hejny
> 
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post117350
> To cancel your subscription to this discussion, please e-mail general-qnx4-unsubscribe@community.qnx.com

Re: virtual circuit close timeout  
Hi,

I just tried it on virtual machines and at first i was not able to reproduce it as well.
I did 'slay -n2 -sstop Fsys.eide' and 'sin ne' did not block.

So i tried to block any process on remote Fsys, e.g. 'ls //2'.
The 'ls' blocked now and then i tried 'sin ne' and it blocked and when i tried to ctrl+c the 'sin ne' it introduced the 
described problem.

stepan hejny
Re: virtual circuit close timeout  
Hi Stepan,

I'll try with ls utility.

Respectfully,
Oleg

23 янв. 2017 г., в 20:15:46, s h <community-noreply@qnx.com> написал:

> Hi,
> 
> I just tried it on virtual machines and at first i was not able to reproduce it as well.
> I did 'slay -n2 -sstop Fsys.eide' and 'sin ne' did not block.
> 
> So i tried to block any process on remote Fsys, e.g. 'ls //2'.
> The 'ls' blocked now and then i tried 'sin ne' and it blocked and when i tried to ctrl+c the 'sin ne' it introduced 
the described problem.
> 
> stepan hejny
> 
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post117370
> To cancel your subscription to this discussion, please e-mail general-qnx4-unsubscribe@community.qnx.com

Re: virtual circuit close timeout  
Hi Stepan,

We’ve reproduced your issue. What is a main problem for you in this situation? Is it behaviour of 'sin ne’ or an 
effect of pressing of Ctr+C? How do you see a solution for the issue? Keep in mind that QNX FLEET extends the QNX IPC to
 the network of tightly coupled QNX microkernels. So I guess the hardware problem in this case may affect on whole QNX 
network.

Respectfully,
Oleg

23 янв. 2017 г., в 20:15, s h <community-noreply@qnx.com> написал:

> Hi,
> 
> I just tried it on virtual machines and at first i was not able to reproduce it as well.
> I did 'slay -n2 -sstop Fsys.eide' and 'sin ne' did not block.
> 
> So i tried to block any process on remote Fsys, e.g. 'ls //2'.
> The 'ls' blocked now and then i tried 'sin ne' and it blocked and when i tried to ctrl+c the 'sin ne' it introduced 
the described problem.
> 
> stepan hejny
> 
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post117370
> To cancel your subscription to this discussion, please e-mail general-qnx4-unsubscribe@community.qnx.com

Re: virtual circuit close timeout  
Hi Oleg,

Our main problem is that local machine is unable to spawn new processes while this timeout is in effect.

And for solution,
we understand that there is a lock in kernel which is engaged while process is terminating (in this case virtual 
circuit).
Is it possible to relax this lock to keep all the stuff safe and at the same time allow spawning of new processes?

Thank you,
stepan hejny
Re: virtual circuit close timeout  
Hi Stepan,

Sorry for the delay with my answer. As I understand correctly, the problem occurs when you press Ctrl+C. Why do you use 
this combination of keys in this situation when you know that this causes the timeout? Will be solution for you the 
updated sin which blocks Ctrl+C signal?

Respectfully,
Oleg

30 янв. 2017 г., в 13:26:20, s h <community-noreply@qnx.com> написал:

> Hi Oleg,
> 
> Our main problem is that local machine is unable to spawn new processes while this timeout is in effect.
> 
> And for solution,
> we understand that there is a lock in kernel which is engaged while process is terminating (in this case virtual 
circuit).
> Is it possible to relax this lock to keep all the stuff safe and at the same time allow spawning of new processes?
> 
> Thank you,
> stepan hejny
> 
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post117383
> To cancel your subscription to this discussion, please e-mail general-qnx4-unsubscribe@community.qnx.com

Re: virtual circuit close timeout  
Hi Oleg,

Oh, the example with sin is just to reproduce the problem easily.
However if this problem is only when breaking sin then sure, we can workaround it.

We have a main and backup system, whatever happens on either one of them must not cause problem for the other. Currently
 communication between the systems is FLEET. If suitable solution is not found we may have to rewrite the communication 
to tcp.

Thank you,
stepan hejny
Re: virtual circuit close timeout  
Hi Stepan,

As I understand the problem occurs after pressing Ctrl+C. Or does it occur any other way also?

Respectfully,
Oleg

8 февр. 2017 г., в 13:42, s h <community-noreply@qnx.com> написал:

> Hi Oleg,
> 
> Oh, the example with sin is just to reproduce the problem easily.
> However if this problem is only when breaking sin then sure, we can workaround it.
> 
> We have a main and backup system, whatever happens on either one of them must not cause problem for the other. 
Currently communication between the systems is FLEET. If suitable solution is not found we may have to rewrite the 
communication to tcp.
> 
> Thank you,
> stepan hejny
> 
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post117402
> To cancel your subscription to this discussion, please e-mail general-qnx4-unsubscribe@community.qnx.com

Re: virtual circuit close timeout  
Hi Oleg,

Yes, as far as i know it happens only when interrupt signal is sent on a process which have virtual circuit in a special
 state.

stepan hejny
Re: virtual circuit close timeout  
Hi Stepan,

Please find attached the sin utility which blocks SIGINT in case of sin ne. I guess you can use this as a workaround.

Respectfully,
Oleg

9 февр. 2017 г., в 19:42:41, s h <community-noreply@qnx.com> написал:

> Hi Oleg,
> 
> Yes, as far as i know it happens only when interrupt signal is sent on a process which have virtual circuit in a 
special state.
> 
> stepan hejny
> 
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post117408
> To cancel your subscription to this discussion, please e-mail general-qnx4-unsubscribe@community.qnx.com
Attachment: Compressed file sin_20170214.tar 45.5 KB
Re: virtual circuit close timeout  
Hi Oleg,

Unfortunately the way we use 'sin ne' is this:
we have a script which is run periodically by cron, this script is checking system and network health and logs this 
information for later examination if needed.
This script executes 'sin ne' and if the sin does not end within a timeout, then a signal is sent to it (does not have 
to be sigint).

Also we found that 'cat' utility also does that and 'wd' watcom debugger too.
And there are probably many other cases.

It is just easy to test it using 'sin ne' and ctrl+c.

Thank you,
stepan hejny