Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - QNX 7 Ramdisk Caching Question: (6 Items)
   
QNX 7 Ramdisk Caching Question  
I have a question about how devb-ram and caching interact with the power safe filesystem.

Here's some background:
On our 6.3.2 systems we create a RAM drive with the following commands:
devb-ram ram capacity=100000 &
mount -t qnx4 /dev/hd1t77 /fs/ram

to create a 50 meg RAM drive where we write run-time log files (not intended to survive reboot obviously). When our 
applications write to a log file on the RAM drive it gets placed there immediately (seen via editing / catting the file)
.

Now on QNX 7 after reading this
http://www.qnx.com/developers/docs/7.0.0/#com.qnx.doc.neutrino.utilities/topic/d/devb-ram.html
we are creating a RAM drive with the following commands:
devb-ram ram capacity=1 blk ramdisk=50m,cache=512k,vnode=256 &
mkqnx6fs -q /dev/ram0
mount -t qnx6 /dev/ram0 /fs/ram

However now when our apps write to the log files there is often nothing there immediately. It seems to take a lot of 
data pushed to the file in order to see earlier writes appear which makes the log file unreliable because stale data may
 not be written ever (even killing the app doesn't flush writes to the file).

So some kind of caching is now occurring which I want to eliminate. Is it the cache=512 (that says it's supposed to be 
the min to io-blk) or is it something in the power safe file system that I need to enable for the RAM drive?

Note that our apps don't call flush() (either in 6.3.2 or 7.0) after doing a write so it's not related to that or I hope
 QNX 7 doesn't require a flush() for every write to get immediate writes.

TIA,

Tim
RE: QNX 7 Ramdisk Caching Question  
Tim,

devb-ram looks and behaves like a real disk, as far as the io-blk layer and filesystem are concerned.  The io-blk 
internal ramdisk which you are now using, is understood by the io-blk caching layer to be a transient RAM disk, and some
 optimizations are made.  These include deferring I/O to the "disk" as long as possible.

With that said, if you write to a file, and then read the file back, you should absolutely see the data that you wrote, 
immediately.  The power-safe filesystem itself still does not realize that the underlying device is a RAM disk, and 
doesn't take any special action.

David
________________________________________
From: Tim Sowden(deleted) [community-noreply@qnx.com]
Sent: Friday, May 18, 2018 5:14 PM
To: general-filesystems
Subject: QNX 7 Ramdisk Caching Question

I have a question about how devb-ram and caching interact with the power safe filesystem.

Here's some background:
On our 6.3.2 systems we create a RAM drive with the following commands:
devb-ram ram capacity=100000 &
mount -t qnx4 /dev/hd1t77 /fs/ram

to create a 50 meg RAM drive where we write run-time log files (not intended to survive reboot obviously). When our 
applications write to a log file on the RAM drive it gets placed there immediately (seen via editing / catting the file)
.

Now on QNX 7 after reading this
http://www.qnx.com/developers/docs/7.0.0/#com.qnx.doc.neutrino.utilities/topic/d/devb-ram.html
we are creating a RAM drive with the following commands:
devb-ram ram capacity=1 blk ramdisk=50m,cache=512k,vnode=256 &
mkqnx6fs -q /dev/ram0
mount -t qnx6 /dev/ram0 /fs/ram

However now when our apps write to the log files there is often nothing there immediately. It seems to take a lot of 
data pushed to the file in order to see earlier writes appear which makes the log file unreliable because stale data may
 not be written ever (even killing the app doesn't flush writes to the file).

So some kind of caching is now occurring which I want to eliminate. Is it the cache=512 (that says it's supposed to be 
the min to io-blk) or is it something in the power safe file system that I need to enable for the RAM drive?

Note that our apps don't call flush() (either in 6.3.2 or 7.0) after doing a write so it's not related to that or I hope
 QNX 7 doesn't require a flush() for every write to get immediate writes.

TIA,

Tim



_______________________________________________

General
http://community.qnx.com/sf/go/post118831
To cancel your subscription to this discussion, please e-mail general-filesystems-unsubscribe@community.qnx.com
Re: RE: QNX 7 Ramdisk Caching Question  
David,

Thanks for the fast reply!

>  These include deferring I/O to the "disk" as long as possible.

Is there a way to turn this off? I'd like to get the same behavior I have under 6.3.2. I only used all the extra options
 in the wiki page because it said it was the fastest way to write data. I don't mind slightly slower writes (it is RAM 
after all so it's already lightning fast) if they occur immediately.

I can 100% reliably produce the problem where data is not available in the RAM log.
Normally I start my app as ./myApp >/fs/ram/myLogFile to redirect stdout to log file.
If I just start ./myApp I see everything printed immediately as the code executes.
If I do the redirect to the log file on the RAM disk I do not get anything in the file until a certain amount of data is
 logged. It appears at a glance to be probably pretty close to that 512 value (bytes?). So roughly every 512 characters 
sent to the file tends to get the next block write to occur to update the file.

So would changing that cache size from 512 matter or is there some other option to look at?

Tim

> Tim,
> 
> devb-ram looks and behaves like a real disk, as far as the io-blk layer and 
> filesystem are concerned.  The io-blk internal ramdisk which you are now using
> , is understood by the io-blk caching layer to be a transient RAM disk, and 
> some optimizations are made.  These include deferring I/O to the "disk" as 
> long as possible.
> 
> With that said, if you write to a file, and then read the file back, you 
> should absolutely see the data that you wrote, immediately.  The power-safe 
> filesystem itself still does not realize that the underlying device is a RAM 
> disk, and doesn't take any special action.
> 
> David
Re: QNX 7 Ramdisk Caching Question  
Tim,

That sounds more like fwrite buffering, so MyApp isn't even sending the data to the FS.  The redirect causes studio to 
become fully buffered, which means it will only flush to the filesystem once the internal stream buffer is full, or an 
fflush() is called.   This is POSIX behaviour so will not have changed since 6.3.2.   I'll check if the buffer size 
changed between that version and 7.0.

If you can change the code, adding

setvbuf(stdout, NULL, _IOLBF);

In your program startup will force stdout to flush after every newline, instead of when the buffer is full.

David

Sent from my BlackBerry - the most secure mobile device
From: community-noreply@qnx.com
Sent: May 18, 2018 5:27 PM
To: general-filesystems@community.qnx.com
Reply-to: general-filesystems@community.qnx.com
Subject: Re: RE: QNX 7 Ramdisk Caching Question


David,

Thanks for the fast reply!

>  These include deferring I/O to the "disk" as long as possible.

Is there a way to turn this off? I'd like to get the same behavior I have under 6.3.2. I only used all the extra options
 in the wiki page because it said it was the fastest way to write data. I don't mind slightly slower writes (it is RAM 
after all so it's already lightning fast) if they occur immediately.

I can 100% reliably produce the problem where data is not available in the RAM log.
Normally I start my app as ./myApp >/fs/ram/myLogFile to redirect stdout to log file.
If I just start ./myApp I see everything printed immediately as the code executes.
If I do the redirect to the log file on the RAM disk I do not get anything in the file until a certain amount of data is
 logged. It appears at a glance to be probably pretty close to that 512 value (bytes?). So roughly every 512 characters 
sent to the file tends to get the next block write to occur to update the file.

So would changing that cache size from 512 matter or is there some other option to look at?

Tim

> Tim,
>
> devb-ram looks and behaves like a real disk, as far as the io-blk layer and
> filesystem are concerned.  The io-blk internal ramdisk which you are now using
> , is understood by the io-blk caching layer to be a transient RAM disk, and
> some optimizations are made.  These include deferring I/O to the "disk" as
> long as possible.
>
> With that said, if you write to a file, and then read the file back, you
> should absolutely see the data that you wrote, immediately.  The power-safe
> filesystem itself still does not realize that the underlying device is a RAM
> disk, and doesn't take any special action.
>
> David




_______________________________________________

General
http://community.qnx.com/sf/go/post118833
To cancel your subscription to this discussion, please e-mail general-filesystems-unsubscribe@community.qnx.com<mailto:
general-filesystems-unsubscribe@community.qnx.com>;
Attachment: HTML sf-attachment-mime36607 6.39 KB
RE: QNX 7 Ramdisk Caching Question  
The stdio buffer size is 1024 bytes, and hasn't changed since at least 6.3.0.

________________________________
From: David Sarrazin
Sent: Friday, May 18, 2018 6:03 PM
To: general-filesystems
Subject: Re: QNX 7 Ramdisk Caching Question

Tim,

That sounds more like fwrite buffering, so MyApp isn't even sending the data to the FS.  The redirect causes studio to 
become fully buffered, which means it will only flush to the filesystem once the internal stream buffer is full, or an 
fflush() is called.   This is POSIX behaviour so will not have changed since 6.3.2.   I'll check if the buffer size 
changed between that version and 7.0.

If you can change the code, adding

setvbuf(stdout, NULL, _IOLBF);

In your program startup will force stdout to flush after every newline, instead of when the buffer is full.

David

Sent from my BlackBerry - the most secure mobile device
From: community-noreply@qnx.com
Sent: May 18, 2018 5:27 PM
To: general-filesystems@community.qnx.com
Reply-to: general-filesystems@community.qnx.com
Subject: Re: RE: QNX 7 Ramdisk Caching Question


David,

Thanks for the fast reply!

>  These include deferring I/O to the "disk" as long as possible.

Is there a way to turn this off? I'd like to get the same behavior I have under 6.3.2. I only used all the extra options
 in the wiki page because it said it was the fastest way to write data. I don't mind slightly slower writes (it is RAM 
after all so it's already lightning fast) if they occur immediately.

I can 100% reliably produce the problem where data is not available in the RAM log.
Normally I start my app as ./myApp >/fs/ram/myLogFile to redirect stdout to log file.
If I just start ./myApp I see everything printed immediately as the code executes.
If I do the redirect to the log file on the RAM disk I do not get anything in the file until a certain amount of data is
 logged. It appears at a glance to be probably pretty close to that 512 value (bytes?). So roughly every 512 characters 
sent to the file tends to get the next block write to occur to update the file.

So would changing that cache size from 512 matter or is there some other option to look at?

Tim

> Tim,
>
> devb-ram looks and behaves like a real disk, as far as the io-blk layer and
> filesystem are concerned.  The io-blk internal ramdisk which you are now using
> , is understood by the io-blk caching layer to be a transient RAM disk, and
> some optimizations are made.  These include deferring I/O to the "disk" as
> long as possible.
>
> With that said, if you write to a file, and then read the file back, you
> should absolutely see the data that you wrote, immediately.  The power-safe
> filesystem itself still does not realize that the underlying device is a RAM
> disk, and doesn't take any special action.
>
> David




_______________________________________________

General
http://community.qnx.com/sf/go/post118833
To cancel your subscription to this discussion, please e-mail general-filesystems-unsubscribe@community.qnx.com<mailto:
general-filesystems-unsubscribe@community.qnx.com>;
Attachment: HTML sf-attachment-mime36610 6.81 KB
Re: RE: QNX 7 Ramdisk Caching Question  
David,

Thanks for looking into the buffer size.

Maybe the size hasn't changed but the write behavior has in some subtle way. Data would not get hung up in the stdio 
buffer forever and never flush. At some point it was getting written even if it was a few seconds delay. Now it remains 
in there forever until the buffer fills up. We've been using this logging behavior for over 10 years and there is no way
 we wouldn't have noticed it.

Regardless, I'll add that line into our code. It won't be at program start but rather will be in a class that is 
responsible for logging. That may mean it gets set a few dozen times per application due to class instantiation but I 
assume that's not a problem.

Tim