Tim Sowden(deleted)
|
QNX 7 Ramdisk Caching Question
|
Tim Sowden(deleted)
05/18/2018 5:14 PM
post118831
|
QNX 7 Ramdisk Caching Question
I have a question about how devb-ram and caching interact with the power safe filesystem.
Here's some background:
On our 6.3.2 systems we create a RAM drive with the following commands:
devb-ram ram capacity=100000 &
mount -t qnx4 /dev/hd1t77 /fs/ram
to create a 50 meg RAM drive where we write run-time log files (not intended to survive reboot obviously). When our
applications write to a log file on the RAM drive it gets placed there immediately (seen via editing / catting the file)
.
Now on QNX 7 after reading this
http://www.qnx.com/developers/docs/7.0.0/#com.qnx.doc.neutrino.utilities/topic/d/devb-ram.html
we are creating a RAM drive with the following commands:
devb-ram ram capacity=1 blk ramdisk=50m,cache=512k,vnode=256 &
mkqnx6fs -q /dev/ram0
mount -t qnx6 /dev/ram0 /fs/ram
However now when our apps write to the log files there is often nothing there immediately. It seems to take a lot of
data pushed to the file in order to see earlier writes appear which makes the log file unreliable because stale data may
not be written ever (even killing the app doesn't flush writes to the file).
So some kind of caching is now occurring which I want to eliminate. Is it the cache=512 (that says it's supposed to be
the min to io-blk) or is it something in the power safe file system that I need to enable for the RAM drive?
Note that our apps don't call flush() (either in 6.3.2 or 7.0) after doing a write so it's not related to that or I hope
QNX 7 doesn't require a flush() for every write to get immediate writes.
TIA,
Tim
|
|
|
David Sarrazin
|
RE: QNX 7 Ramdisk Caching Question
|
David Sarrazin
05/18/2018 5:27 PM
post118832
|
RE: QNX 7 Ramdisk Caching Question
Tim,
devb-ram looks and behaves like a real disk, as far as the io-blk layer and filesystem are concerned. The io-blk
internal ramdisk which you are now using, is understood by the io-blk caching layer to be a transient RAM disk, and some
optimizations are made. These include deferring I/O to the "disk" as long as possible.
With that said, if you write to a file, and then read the file back, you should absolutely see the data that you wrote,
immediately. The power-safe filesystem itself still does not realize that the underlying device is a RAM disk, and
doesn't take any special action.
David
________________________________________
From: Tim Sowden(deleted) [community-noreply@qnx.com]
Sent: Friday, May 18, 2018 5:14 PM
To: general-filesystems
Subject: QNX 7 Ramdisk Caching Question
I have a question about how devb-ram and caching interact with the power safe filesystem.
Here's some background:
On our 6.3.2 systems we create a RAM drive with the following commands:
devb-ram ram capacity=100000 &
mount -t qnx4 /dev/hd1t77 /fs/ram
to create a 50 meg RAM drive where we write run-time log files (not intended to survive reboot obviously). When our
applications write to a log file on the RAM drive it gets placed there immediately (seen via editing / catting the file)
.
Now on QNX 7 after reading this
http://www.qnx.com/developers/docs/7.0.0/#com.qnx.doc.neutrino.utilities/topic/d/devb-ram.html
we are creating a RAM drive with the following commands:
devb-ram ram capacity=1 blk ramdisk=50m,cache=512k,vnode=256 &
mkqnx6fs -q /dev/ram0
mount -t qnx6 /dev/ram0 /fs/ram
However now when our apps write to the log files there is often nothing there immediately. It seems to take a lot of
data pushed to the file in order to see earlier writes appear which makes the log file unreliable because stale data may
not be written ever (even killing the app doesn't flush writes to the file).
So some kind of caching is now occurring which I want to eliminate. Is it the cache=512 (that says it's supposed to be
the min to io-blk) or is it something in the power safe file system that I need to enable for the RAM drive?
Note that our apps don't call flush() (either in 6.3.2 or 7.0) after doing a write so it's not related to that or I hope
QNX 7 doesn't require a flush() for every write to get immediate writes.
TIA,
Tim
_______________________________________________
General
http://community.qnx.com/sf/go/post118831
To cancel your subscription to this discussion, please e-mail general-filesystems-unsubscribe@community.qnx.com
|
|
|
Tim Sowden(deleted)
|
Re: RE: QNX 7 Ramdisk Caching Question
|
Tim Sowden(deleted)
05/18/2018 5:46 PM
post118833
|
Re: RE: QNX 7 Ramdisk Caching Question
David,
Thanks for the fast reply!
> These include deferring I/O to the "disk" as long as possible.
Is there a way to turn this off? I'd like to get the same behavior I have under 6.3.2. I only used all the extra options
in the wiki page because it said it was the fastest way to write data. I don't mind slightly slower writes (it is RAM
after all so it's already lightning fast) if they occur immediately.
I can 100% reliably produce the problem where data is not available in the RAM log.
Normally I start my app as ./myApp >/fs/ram/myLogFile to redirect stdout to log file.
If I just start ./myApp I see everything printed immediately as the code executes.
If I do the redirect to the log file on the RAM disk I do not get anything in the file until a certain amount of data is
logged. It appears at a glance to be probably pretty close to that 512 value (bytes?). So roughly every 512 characters
sent to the file tends to get the next block write to occur to update the file.
So would changing that cache size from 512 matter or is there some other option to look at?
Tim
> Tim,
>
> devb-ram looks and behaves like a real disk, as far as the io-blk layer and
> filesystem are concerned. The io-blk internal ramdisk which you are now using
> , is understood by the io-blk caching layer to be a transient RAM disk, and
> some optimizations are made. These include deferring I/O to the "disk" as
> long as possible.
>
> With that said, if you write to a file, and then read the file back, you
> should absolutely see the data that you wrote, immediately. The power-safe
> filesystem itself still does not realize that the underlying device is a RAM
> disk, and doesn't take any special action.
>
> David
|
|
|
David Sarrazin
|
Re: QNX 7 Ramdisk Caching Question
|
David Sarrazin
05/18/2018 6:22 PM
post118834
|
Re: QNX 7 Ramdisk Caching Question
Tim,
That sounds more like fwrite buffering, so MyApp isn't even sending the data to the FS. The redirect causes studio to
become fully buffered, which means it will only flush to the filesystem once the internal stream buffer is full, or an
fflush() is called. This is POSIX behaviour so will not have changed since 6.3.2. I'll check if the buffer size
changed between that version and 7.0.
If you can change the code, adding
setvbuf(stdout, NULL, _IOLBF);
In your program startup will force stdout to flush after every newline, instead of when the buffer is full.
David
Sent from my BlackBerry - the most secure mobile device
From: community-noreply@qnx.com
Sent: May 18, 2018 5:27 PM
To: general-filesystems@community.qnx.com
Reply-to: general-filesystems@community.qnx.com
Subject: Re: RE: QNX 7 Ramdisk Caching Question
David,
Thanks for the fast reply!
> These include deferring I/O to the "disk" as long as possible.
Is there a way to turn this off? I'd like to get the same behavior I have under 6.3.2. I only used all the extra options
in the wiki page because it said it was the fastest way to write data. I don't mind slightly slower writes (it is RAM
after all so it's already lightning fast) if they occur immediately.
I can 100% reliably produce the problem where data is not available in the RAM log.
Normally I start my app as ./myApp >/fs/ram/myLogFile to redirect stdout to log file.
If I just start ./myApp I see everything printed immediately as the code executes.
If I do the redirect to the log file on the RAM disk I do not get anything in the file until a certain amount of data is
logged. It appears at a glance to be probably pretty close to that 512 value (bytes?). So roughly every 512 characters
sent to the file tends to get the next block write to occur to update the file.
So would changing that cache size from 512 matter or is there some other option to look at?
Tim
> Tim,
>
> devb-ram looks and behaves like a real disk, as far as the io-blk layer and
> filesystem are concerned. The io-blk internal ramdisk which you are now using
> , is understood by the io-blk caching layer to be a transient RAM disk, and
> some optimizations are made. These include deferring I/O to the "disk" as
> long as possible.
>
> With that said, if you write to a file, and then read the file back, you
> should absolutely see the data that you wrote, immediately. The power-safe
> filesystem itself still does not realize that the underlying device is a RAM
> disk, and doesn't take any special action.
>
> David
_______________________________________________
General
http://community.qnx.com/sf/go/post118833
To cancel your subscription to this discussion, please e-mail general-filesystems-unsubscribe@community.qnx.com<mailto:
general-filesystems-unsubscribe@community.qnx.com>
|
|
|
David Sarrazin
|
RE: QNX 7 Ramdisk Caching Question
|
David Sarrazin
05/19/2018 2:26 PM
post118835
|
RE: QNX 7 Ramdisk Caching Question
The stdio buffer size is 1024 bytes, and hasn't changed since at least 6.3.0.
________________________________
From: David Sarrazin
Sent: Friday, May 18, 2018 6:03 PM
To: general-filesystems
Subject: Re: QNX 7 Ramdisk Caching Question
Tim,
That sounds more like fwrite buffering, so MyApp isn't even sending the data to the FS. The redirect causes studio to
become fully buffered, which means it will only flush to the filesystem once the internal stream buffer is full, or an
fflush() is called. This is POSIX behaviour so will not have changed since 6.3.2. I'll check if the buffer size
changed between that version and 7.0.
If you can change the code, adding
setvbuf(stdout, NULL, _IOLBF);
In your program startup will force stdout to flush after every newline, instead of when the buffer is full.
David
Sent from my BlackBerry - the most secure mobile device
From: community-noreply@qnx.com
Sent: May 18, 2018 5:27 PM
To: general-filesystems@community.qnx.com
Reply-to: general-filesystems@community.qnx.com
Subject: Re: RE: QNX 7 Ramdisk Caching Question
David,
Thanks for the fast reply!
> These include deferring I/O to the "disk" as long as possible.
Is there a way to turn this off? I'd like to get the same behavior I have under 6.3.2. I only used all the extra options
in the wiki page because it said it was the fastest way to write data. I don't mind slightly slower writes (it is RAM
after all so it's already lightning fast) if they occur immediately.
I can 100% reliably produce the problem where data is not available in the RAM log.
Normally I start my app as ./myApp >/fs/ram/myLogFile to redirect stdout to log file.
If I just start ./myApp I see everything printed immediately as the code executes.
If I do the redirect to the log file on the RAM disk I do not get anything in the file until a certain amount of data is
logged. It appears at a glance to be probably pretty close to that 512 value (bytes?). So roughly every 512 characters
sent to the file tends to get the next block write to occur to update the file.
So would changing that cache size from 512 matter or is there some other option to look at?
Tim
> Tim,
>
> devb-ram looks and behaves like a real disk, as far as the io-blk layer and
> filesystem are concerned. The io-blk internal ramdisk which you are now using
> , is understood by the io-blk caching layer to be a transient RAM disk, and
> some optimizations are made. These include deferring I/O to the "disk" as
> long as possible.
>
> With that said, if you write to a file, and then read the file back, you
> should absolutely see the data that you wrote, immediately. The power-safe
> filesystem itself still does not realize that the underlying device is a RAM
> disk, and doesn't take any special action.
>
> David
_______________________________________________
General
http://community.qnx.com/sf/go/post118833
To cancel your subscription to this discussion, please e-mail general-filesystems-unsubscribe@community.qnx.com<mailto:
general-filesystems-unsubscribe@community.qnx.com>
|
|
|
Tim Sowden(deleted)
|
Re: RE: QNX 7 Ramdisk Caching Question
|
Tim Sowden(deleted)
05/19/2018 3:21 PM
post118836
|
Re: RE: QNX 7 Ramdisk Caching Question
David,
Thanks for looking into the buffer size.
Maybe the size hasn't changed but the write behavior has in some subtle way. Data would not get hung up in the stdio
buffer forever and never flush. At some point it was getting written even if it was a few seconds delay. Now it remains
in there forever until the buffer fills up. We've been using this logging behavior for over 10 years and there is no way
we wouldn't have noticed it.
Regardless, I'll add that line into our code. It won't be at program start but rather will be in a class that is
responsible for logging. That may mean it gets set a few dozen times per application due to class instantiation but I
assume that's not a problem.
Tim
|
|
|
|