RLE Compression for File Formats #95

Merged
kirsle merged 6 commits from rle-compression into master 2024-05-24 23:48:00 +00:00
Owner

This adds a new RLEAccessor type of chunk accessor which stores drawing data with RLE (Run Length Encoding) compression.

On average it results in file size savings of 90+% especially for levels having large blocks of colors.

This adds a new RLEAccessor type of chunk accessor which stores drawing data with RLE (Run Length Encoding) compression. On average it results in file size savings of 90+% especially for levels having large blocks of colors.
kirsle added the
enhancement
label 2024-05-24 23:28:11 +00:00
kirsle added 5 commits 2024-05-24 23:28:12 +00:00
Finally add a second option for Chunk MapAccessor implementation besides the
MapAccessor. The RLEAccessor is basically a MapAccessor that will compress
your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP
file.

This slashes the file sizes of most levels:

* Shapeshifter: 21.8 MB -> 8.1 MB
* Jungle: 10.4 MB -> 4.1 MB
* Zoo: 2.8 MB -> 1.3 MB

Implementation details:

* The RLE binary format for Chunks is a stream of Uvarint pairs storing the
  palette index number and the number of pixels to repeat it (along the Y,X
  axis of the chunk).
    * Null colors are represented by a Uvarint that decodes to 0xFFFF
      or 65535 in decimal.
    * Gameplay logic currently limits maps to 256 colors.
* The default for newly created chunks in-game will be RLE by default.
* Its in-memory representation is still a MapAccessor (a map of absolute
  world coordinates to palette index).
* The game can still open and play legacy MapAccessor maps.
* On save in the editor, the game will upgrade/convert MapAccessor chunks over
  to RLEAccessors, improving on your level's file size with a simple re-save.

Current Bugs

* On every re-save to RLE, one pixel is lost in the bottom-right corner of
  each chunk. Each subsequent re-save loses one more pixel to the left, so what
  starts as a single pixel per chunk slowly evolves into a horizontal line.
* Some pixels smear vertically as well.
* Off-by-negative-one errors when some chunks Iter() their pixels but compute
  a relative coordinate of (-1,0)! Some mismatch between the stored world coords
  of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker:
  certain combinations of chunk coord/abs coord.

To Do

* The `doodad touch` command should re-save existing levels to upgrade them.
Levels can now be converted to RLE encoded chunk accessors and be re-saved
continuously without any loss of information.

Off-by-one errors resolved:

* The rle.NewGrid() was adding a +1 everywhere making the 2D grids have 129
  elements to a side for a 128 chunk size.
* In rle.Decompress() the cursor value and translation to X,Y coordinates is
  fixed to avoid a pixel going missing at the end of the first row (128,0)
* The abs.X-- hack in UnmarshalBinary is no longer needed to prevent the
  chunks from scooting a pixel to the right on every save.

Doodad tool updates:

* Remove unused CLI flags in `doodad resave` (actors, chunks, script,
  attachment, verbose) and add a `--output` flag to save to a different file
  name to the original.
* Update `doodad show` to allow debugging of RLE compressed chunks:
    * CLI flag `--chunk=1,2` to specify a single chunk coordinate to debug
    * CLI flag `--visualize-rle` will Visualize() RLE compressed chunks in
      their 2D grid form in your terminal window (VERY noisy for large
      levels! Use the --chunk option to narrow to one chunk).

Bug fixes and misc changes:

* Chunk.Usage() to return a better percentage of chunk utilization.
* Chunker.ChunkFromZipfile() was split out into two functions:
    * RawChunkFromZipfile retrieves the raw bytes of the chunk as well as the
      file extension discovered (.bin or .json) so the caller can interpret
      the bytes correctly.
    * ChunkFromZipfile calls the former function and then depending on file
      extension, unmarshals from binary or json.
    * The Raw function enables the `doodad show` command to debug and visualize
      the raw contents of the RLE compressed chunks.
* Updated the Visualize() function for the RLE encoder: instead of converting
  palette indexes to hex (0-F) which would begin causing problems for palette
  indexes above 16 (as they would use two+ characters), indexes are mapped to
  a wider range of symbols (0-9A-Z) and roll over if you have more than 36
  colors on your level. This at least keeps the Visualize() grid an easy to
  read 128x128 characters in your terminal.
* For the doodad tool: skip the assets embed folder, the doodad binary doesn't
  need to include all the game's doodads/levelpacks/etc. and can save on file
  size.
* In `doodad resave`, .doodad files with Vacuum() and upgrade their chunker from
  the MapAccessor to the RLEAccessor.
* Fix a rare concurrent map read/write error in OptimizeChunkerAccessors.
* Add documentation for the game's file formats and RLE encoding
* Remove the never-used GridType Chunk Accessor constant
kirsle added 1 commit 2024-05-24 23:43:21 +00:00
kirsle merged commit 57b757a378 into master 2024-05-24 23:48:00 +00:00
kirsle deleted branch rle-compression 2024-05-24 23:48:00 +00:00
Sign in to join this conversation.
No reviewers
No Milestone
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: SketchyMaze/doodle#95
No description provided.