Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - What's the LZO block size used in mkifs?: (6 Items)
   
What's the LZO block size used in mkifs?  
I guess it's 64KB since I can compress same data to generate same block size list as in IFS's compressed imagefs.

I have added "[compress=LZO]" to my build file of cause.

The layout of compressed blocks are in: [block_size] [compressed_data], and block_size is a 16-bit big-endian number.

Here is the problem: if len(lzo.compress(uncompressed_data_in_64KB)) >= 64 KB, then some data are lost. since block_size
 is a 16-bit big-endian number.

A python demo code and ifs image are attached to reproduce this issue.

Attachment: Text ifs_block_size_demo.7z 2.53 MB
Re: What's the LZO block size used in mkifs?  
64kiB is the _maximum_ block size used. Theoretically, mkifs should check the output size of each LZO block right after 
the compression step, and if the output length exceeds 0x10000, reduce the input length and retry. Hence, the resulting 
IFS should never contain an LZO block larger than 64kiB and the 16-bit block size should always be sufficient.

Is this an actual issue with a compressed IFS generated by mkifs, or merely theoretical? If it's an actual issue, can 
you provide a test case (build file + all files included in image)?

Kind regards,
Thomas
Re: What's the LZO block size used in mkifs?  
> 64kiB is the _maximum_ block size used. Theoretically, mkifs should check the 
> output size of each LZO block right after the compression step, and if the 
> output length exceeds 0x10000, reduce the input length and retry. Hence, the 
> resulting IFS should never contain an LZO block larger than 64kiB and the 16-
> bit block size should always be sufficient.
> 
> Is this an actual issue with a compressed IFS generated by mkifs, or merely 
> theoretical? If it's an actual issue, can you provide a test case (build file 
> + all files included in image)?
> 
> Kind regards,
> Thomas

Thanks for your reply! I can reproduce some lengh list of compressed blocks now.
Yes. I checked your 'theoretical reducing', that's true. Here is what mkifs doing:
> block = lzo.compress(bytes(data[i : i + block_size]), level, False)
> if len(block) >= max_block:
>     bs = block_size - int(max_block / 16)
>     block = lzo.compress(bytes(data[i : i + bs]), level, False)
>     blocks.append(struct.pack(head_fmt, len(block)) + block)
>     block = lzo.compress(bytes(data[i + bs: i + block_size]), level, False)
So this is just a theoretical issue.
And I can't post whole build and image files here since they are commercial productions.
Thanks again!
Re: What's the LZO block size used in mkifs?  
Last time I checked the the lzo and ucl compression in an ifs image use a fixed input block size of 16KB.
As you state the output size is then stored in the first 2 bytes of the block.

This compression is mainly used in the primary IFS which is normally used as a spring board.
i.e. once you have a primary ifs loaded you have a system and can then use a multithreaded env to load secondary/
tertiary IFS images and for those images you can use whatever encryption/signing/compression algorithm you want.
Re: What's the LZO block size used in mkifs?  
Actually I miss remembered, it's actually 64KB.
Re: What's the LZO block size used in mkifs?  
The input block size for LZO compression is 64KiB.