Size: 2734
Comment: Links to current coding work
|
Size: 3281
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
=== This page contains list of patches and other relatd code as it is being developed for Compressed Caching (for 2.6.x kernels) project. You can always find most up-to-date progress of code at Project CVS. The code is presently very raw but its helping me getting more experience with VMM code and how things are to be done :) === | [:NitinGupta:Nitin Gupta] [[MailTo(nitingupta910 AT gmail DOT com)]] |
Line 3: | Line 5: |
=== Kernel changes to support Compressed Caching: === * [attachment:toy-cc-2.6.16-rc4.diff Toy ccache patch]: These are few lines I added to 2.6.16-rc4 while going through VMM code. Just some printk()s to simply highlight some kernel entry points for compressed caching work. |
This page contains some prototype code developed for CompressedCaching project. After seriously insufficient testing, each of these components were eventually merged into final implementation on 2.6.x kernels. ---- || [http://linuxcompressed.sourceforge.net/ Project Home Page] || CompressedCaching || [http://sourceforge.net/project/showfiles.php?group_id=13472 Patches] || |
Line 6: | Line 9: |
* [attachment:patch-cc-2.6.16-radix-replace-stable.diff patch-cc-2.6.16-radix-replace-stable]: Replace original page (for now, only clean page cache pages) with a 'chunk head' when it is to be freed under memory pressure and simply store original page uncompressed. When page cache lookup is performed, again replace the 'chunk head' with original page. This patch uses simplified (and inefficient) locking in page cache lookup functions to make it stable for now. | === Compression algorithms to kernel mode: === Kernel module to test de/compression algorithms (WKdm, WK4x4, LZO): attachment:compress-test.tar.gz |
Line 8: | Line 12: |
* [attachment:patch-cc-2.6.16-better-locking-unstable.diff patch-cc-2.6.16-better-locking-unstable]: This was an attempt to get a better (more efficint) locking in page cache lookup functions but it is not quite as stable as previous simplified patch. It causes apps to freeze as swap usage increase. | There are basically three main algorithms that are well studied w.r.t compressed caching by previous works -- WKdm, WK4x4, LZO. Of these, WKdm, WK4x4 are designed to handle anon pages (non filesystem pages) while LZO is more suitable for filesystem data. (Also, in general, compression speed is in order: WKdm > WK4x4 > LZO, while compression factor order is, in general, reverse). Now, all three algos are ported to kernel space - WKdm, WK4x4 and LZO. You can test them all using this module. It creates 3 /proc entries: /proc/compress-test/{compress, decompress, algo_idx} as described below: (for some detail see README with this module) ''' In short:''' __''Write to /proc/compress-test entries:''__ 1. ''compress'': compress data witten to it and store in internal buffer. 2. ''algo_idx'': write index of algo you want to test (0: WKdm, 1: WK4x4, 2: LZO) __''Read from /proc/compress-test entries:''__ 1. ''compress'': show original and compressed size (TODO: add other stats like time taken too) 2. ''decompress'': decompress compressed data stored in internal buffer. 3. ''algo_idx'': shows list of algos supported with their index. Also commited all these algos to GIT repo: git://dev.laptop.org/projects/linux-mm-cc |
Line 11: | Line 37: |
=== Compression algorithms to kernel mode: === There are basically three main algorithms that are well studied w.r.t compressed caching by previous works -- WKdm, WK4x4, LZO. |
=== Compression Structure Implementation: === attachment:storage-test.tar.gz : This module implements compression structure as described on CompressedCaching. |
Line 14: | Line 40: |
Of these, WKdm, WK4x4 are designed to handle anon pages (non filesystem pages) while LZO is more suitable for filesystem data. (Also, in general, compression speed is in order: WKdm > WK4x4 > LZO, while compression factor order is, in general, reverse) Currently, I have only ported WKdm to kernel mode: ( attachment:compress%2Dtest.tar.gz ): This is a kernel module to test port of WKdm algorithm to kernel mode. It creates two /proc entries: /proc/{compress, decompress}. |
* The storage begins as a single page. As you add pages to it, it expands till it reaches its max limit. As you take out pages, it shrinks, freeing up pages when it has no chunks left. * Adjacent free chunks are merged together. * Each page can be compressed using different algo. |
Line 18: | Line 44: |
Writing a file to /proc/compress compresses the input file (input file must be <= 4K i.e. PAGE_SIZE) and stores it in an internal buffer. Reading /proc/decompress again shows decompressed output. Reading /proc/compress stores statistics (currently only original and compressed size) like de/compression time, dictionary hits (partial matches, exact matches, missses). |
Please see README for usage. |
Line 23: | Line 46: |
This makes it easy to test algotithm by simply comparing original file with /proc/decompress output and statistics can be obtained by reading /proc/compress. This module will serve as simple test-bed when porting other important de/compression algorithms w.r.t compressed caching (WK4x4, LZO). | ''' In short:''' |
Line 25: | Line 48: |
Interface is via /proc: /proc/storage-test/{readpage, writepage, show_structure} 1. ''writepage'': write a page on this to compress and store it in ccache. 2. ''readpage'': write 'id' (see README) of page you want. 3. ''show_structure'': read to show current snapshot of ccache storage. ---- [[BR]] [[BR]] '' There was a university where computer science students could ask programming questions to faculty staff. [[BR]][[BR]] However, before taking their problem and question to the staff, they first had to explain their problem and ask their question to a teddy bear in the lobby. Legend has it that 90% of the students found the answer to their question after putting their problem in words, just by asking it to the teddy bear. [[BR]][[BR]] They never needed to ask the staff, all they needed was to organize their thoughts. '' -- picked up from kernelnewbies.org |
[:NitinGupta:Nitin Gupta]
MailTo(nitingupta910 AT gmail DOT com)
This page contains some prototype code developed for CompressedCaching project. After seriously insufficient testing, each of these components were eventually merged into final implementation on 2.6.x kernels.
[http://linuxcompressed.sourceforge.net/ Project Home Page] |
[http://sourceforge.net/project/showfiles.php?group_id=13472 Patches] |
Compression algorithms to kernel mode:
Kernel module to test de/compression algorithms (WKdm, WK4x4, LZO): attachment:compress-test.tar.gz
There are basically three main algorithms that are well studied w.r.t compressed caching by previous works -- WKdm, WK4x4, LZO.
Of these, WKdm, WK4x4 are designed to handle anon pages (non filesystem pages) while LZO is more suitable for filesystem data. (Also, in general, compression speed is in order: WKdm > WK4x4 > LZO, while compression factor order is, in general, reverse).
Now, all three algos are ported to kernel space - WKdm, WK4x4 and LZO. You can test them all using this module. It creates 3 /proc entries: /proc/compress-test/{compress, decompress, algo_idx} as described below: (for some detail see README with this module)
In short:
Write to /proc/compress-test entries:
1. compress: compress data witten to it and store in internal buffer.
2. algo_idx: write index of algo you want to test (0: WKdm, 1: WK4x4, 2: LZO)
Read from /proc/compress-test entries:
1. compress: show original and compressed size (TODO: add other stats like time taken too)
2. decompress: decompress compressed data stored in internal buffer.
3. algo_idx: shows list of algos supported with their index.
Also commited all these algos to GIT repo: git://dev.laptop.org/projects/linux-mm-cc
Compression Structure Implementation:
attachment:storage-test.tar.gz : This module implements compression structure as described on CompressedCaching.
- The storage begins as a single page. As you add pages to it, it expands till it reaches its max limit. As you take out pages, it shrinks, freeing up pages when it has no chunks left.
- Adjacent free chunks are merged together.
- Each page can be compressed using different algo.
Please see README for usage.
In short:
Interface is via /proc:
/proc/storage-test/{readpage, writepage, show_structure}
1. writepage: write a page on this to compress and store it in ccache.
2. readpage: write 'id' (see README) of page you want.
3. show_structure: read to show current snapshot of ccache storage.
There was a university where computer science students could ask programming questions to faculty staff. BRBR However, before taking their problem and question to the staff, they first had to explain their problem and ask their question to a teddy bear in the lobby. Legend has it that 90% of the students found the answer to their question after putting their problem in words, just by asking it to the teddy bear. BRBR They never needed to ask the staff, all they needed was to organize their thoughts.