RLE Compression for File Formats #95
No reviewers
Labels
No Milestone
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: SketchyMaze/doodle#95
Loading…
Reference in New Issue
There is no content yet.
Delete Branch "rle-compression"
Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?
This adds a new RLEAccessor type of chunk accessor which stores drawing data with RLE (Run Length Encoding) compression.
On average it results in file size savings of 90+% especially for levels having large blocks of colors.
Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.Levels can now be converted to RLE encoded chunk accessors and be re-saved continuously without any loss of information. Off-by-one errors resolved: * The rle.NewGrid() was adding a +1 everywhere making the 2D grids have 129 elements to a side for a 128 chunk size. * In rle.Decompress() the cursor value and translation to X,Y coordinates is fixed to avoid a pixel going missing at the end of the first row (128,0) * The abs.X-- hack in UnmarshalBinary is no longer needed to prevent the chunks from scooting a pixel to the right on every save. Doodad tool updates: * Remove unused CLI flags in `doodad resave` (actors, chunks, script, attachment, verbose) and add a `--output` flag to save to a different file name to the original. * Update `doodad show` to allow debugging of RLE compressed chunks: * CLI flag `--chunk=1,2` to specify a single chunk coordinate to debug * CLI flag `--visualize-rle` will Visualize() RLE compressed chunks in their 2D grid form in your terminal window (VERY noisy for large levels! Use the --chunk option to narrow to one chunk). Bug fixes and misc changes: * Chunk.Usage() to return a better percentage of chunk utilization. * Chunker.ChunkFromZipfile() was split out into two functions: * RawChunkFromZipfile retrieves the raw bytes of the chunk as well as the file extension discovered (.bin or .json) so the caller can interpret the bytes correctly. * ChunkFromZipfile calls the former function and then depending on file extension, unmarshals from binary or json. * The Raw function enables the `doodad show` command to debug and visualize the raw contents of the RLE compressed chunks. * Updated the Visualize() function for the RLE encoder: instead of converting palette indexes to hex (0-F) which would begin causing problems for palette indexes above 16 (as they would use two+ characters), indexes are mapped to a wider range of symbols (0-9A-Z) and roll over if you have more than 36 colors on your level. This at least keeps the Visualize() grid an easy to read 128x128 characters in your terminal.