doodle/pkg/level/chunker_migrate.go

68 lines
1.5 KiB
Go
Raw Permalink Normal View History

(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
package level
import (
"runtime"
"sync"
"git.kirsle.net/SketchyMaze/doodle/pkg/balance"
"git.kirsle.net/SketchyMaze/doodle/pkg/log"
)
/* Functions to migrate Chunkers between different implementations. */
// OptimizeChunkerAccessors will evaluate all of the chunks of your drawing
// and possibly migrate them to a different Accessor implementation when
// saving on disk.
func (c *Chunker) OptimizeChunkerAccessors() {
2024-05-24 23:43:11 +00:00
// Parallelize this with goroutines.
(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
var (
chunks = make(chan *Chunk, len(c.Chunks))
wg sync.WaitGroup
)
for range runtime.NumCPU() {
wg.Add(1)
go func() {
defer wg.Done()
for chunk := range chunks {
var point = chunk.Point
// Upgrade all MapTypes into RLE compressed MapTypes?
if balance.RLEBinaryChunkerEnabled {
if chunk.Type == MapType {
log.Info("Optimizing chunk %s accessor from Map to RLE", point)
ma, _ := chunk.Accessor.(*MapAccessor)
rle := NewRLEAccessor(chunk).FromMapAccessor(ma)
// Lock the chunker for updating.
c.chunkMu.Lock()
(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
c.Chunks[point].Type = RLEType
c.Chunks[point].Accessor = rle
c.chunkMu.Unlock()
(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
}
}
}
}()
}
// Feed it the chunks.
Fix RLE Encoding Off-by-One Errors [PTO] Levels can now be converted to RLE encoded chunk accessors and be re-saved continuously without any loss of information. Off-by-one errors resolved: * The rle.NewGrid() was adding a +1 everywhere making the 2D grids have 129 elements to a side for a 128 chunk size. * In rle.Decompress() the cursor value and translation to X,Y coordinates is fixed to avoid a pixel going missing at the end of the first row (128,0) * The abs.X-- hack in UnmarshalBinary is no longer needed to prevent the chunks from scooting a pixel to the right on every save. Doodad tool updates: * Remove unused CLI flags in `doodad resave` (actors, chunks, script, attachment, verbose) and add a `--output` flag to save to a different file name to the original. * Update `doodad show` to allow debugging of RLE compressed chunks: * CLI flag `--chunk=1,2` to specify a single chunk coordinate to debug * CLI flag `--visualize-rle` will Visualize() RLE compressed chunks in their 2D grid form in your terminal window (VERY noisy for large levels! Use the --chunk option to narrow to one chunk). Bug fixes and misc changes: * Chunk.Usage() to return a better percentage of chunk utilization. * Chunker.ChunkFromZipfile() was split out into two functions: * RawChunkFromZipfile retrieves the raw bytes of the chunk as well as the file extension discovered (.bin or .json) so the caller can interpret the bytes correctly. * ChunkFromZipfile calls the former function and then depending on file extension, unmarshals from binary or json. * The Raw function enables the `doodad show` command to debug and visualize the raw contents of the RLE compressed chunks. * Updated the Visualize() function for the RLE encoder: instead of converting palette indexes to hex (0-F) which would begin causing problems for palette indexes above 16 (as they would use two+ characters), indexes are mapped to a wider range of symbols (0-9A-Z) and roll over if you have more than 36 colors on your level. This at least keeps the Visualize() grid an easy to read 128x128 characters in your terminal.
2024-05-24 20:54:41 +00:00
for point := range c.IterChunks() {
chunk, ok := c.GetChunk(point)
if !ok {
continue
}
(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
chunks <- chunk
}
close(chunks)
wg.Wait()
}
// FromMapAccessor migrates from a MapAccessor to RLE.
func (a *RLEAccessor) FromMapAccessor(ma *MapAccessor) *RLEAccessor {
return &RLEAccessor{
chunk: a.chunk,
acc: ma,
}
}