doodle/pkg/level/chunk_map.go

306 lines
7.4 KiB
Go
Raw Permalink Normal View History

Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
package level
import (
2023-02-18 20:45:36 +00:00
"bytes"
"encoding/binary"
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
"encoding/json"
"errors"
"fmt"
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
"sync"
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
2023-02-18 20:45:36 +00:00
"git.kirsle.net/SketchyMaze/doodle/pkg/balance"
"git.kirsle.net/SketchyMaze/doodle/pkg/log"
"git.kirsle.net/go/render"
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
)
// MapAccessor implements a chunk accessor by using a map of points to their
// palette indexes. This is the simplest accessor and is best for sparse chunks.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
type MapAccessor struct {
(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
chunk *Chunk // Pointer to parent struct, for its Size and Point
2024-05-24 02:15:10 +00:00
grid map[render.Point]*Swatch
mu sync.RWMutex
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
// NewMapAccessor initializes a MapAccessor.
(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
func NewMapAccessor(chunk *Chunk) *MapAccessor {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
return &MapAccessor{
(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
chunk: chunk,
grid: map[render.Point]*Swatch{},
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
(Experimental) Run Length Encoding for Levels Finally add a second option for Chunk MapAccessor implementation besides the MapAccessor. The RLEAccessor is basically a MapAccessor that will compress your drawing with Run Length Encoding (RLE) in the on-disk format in the ZIP file. This slashes the file sizes of most levels: * Shapeshifter: 21.8 MB -> 8.1 MB * Jungle: 10.4 MB -> 4.1 MB * Zoo: 2.8 MB -> 1.3 MB Implementation details: * The RLE binary format for Chunks is a stream of Uvarint pairs storing the palette index number and the number of pixels to repeat it (along the Y,X axis of the chunk). * Null colors are represented by a Uvarint that decodes to 0xFFFF or 65535 in decimal. * Gameplay logic currently limits maps to 256 colors. * The default for newly created chunks in-game will be RLE by default. * Its in-memory representation is still a MapAccessor (a map of absolute world coordinates to palette index). * The game can still open and play legacy MapAccessor maps. * On save in the editor, the game will upgrade/convert MapAccessor chunks over to RLEAccessors, improving on your level's file size with a simple re-save. Current Bugs * On every re-save to RLE, one pixel is lost in the bottom-right corner of each chunk. Each subsequent re-save loses one more pixel to the left, so what starts as a single pixel per chunk slowly evolves into a horizontal line. * Some pixels smear vertically as well. * Off-by-negative-one errors when some chunks Iter() their pixels but compute a relative coordinate of (-1,0)! Some mismatch between the stored world coords of a pixel inside the chunk vs. the chunk's assigned coordinate by the Chunker: certain combinations of chunk coord/abs coord. To Do * The `doodad touch` command should re-save existing levels to upgrade them.
2024-05-24 06:02:01 +00:00
// Reset the MapAccessor.
func (a *MapAccessor) Reset() {
a.grid = map[render.Point]*Swatch{}
2024-05-24 02:15:10 +00:00
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
// Inflate the sparse swatches from their palette indexes.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) Inflate(pal *Palette) error {
for point, swatch := range a.grid {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
if swatch.IsSparse() {
// Replace this with the correct swatch from the palette.
if swatch.paletteIndex >= len(pal.Swatches) {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
return fmt.Errorf("MapAccessor.Inflate: swatch for point %s has paletteIndex %d but palette has only %d colors",
point,
swatch.paletteIndex,
len(pal.Swatches),
)
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
a.mu.Lock()
a.grid[point] = pal.Swatches[swatch.paletteIndex] // <- concurrent write
a.mu.Unlock()
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
}
return nil
}
// Len returns the current size of the map, or number of pixels registered.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) Len() int {
a.mu.RLock()
defer a.mu.RUnlock()
return len(a.grid)
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
// IterViewport returns a channel to loop over pixels in the viewport.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) IterViewport(viewport render.Rect) <-chan Pixel {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
pipe := make(chan Pixel)
go func() {
for px := range a.Iter() {
if px.Point().Inside(viewport) {
pipe <- px
}
}
close(pipe)
}()
return pipe
}
// Iter returns a channel to loop over all points in this chunk.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) Iter() <-chan Pixel {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
pipe := make(chan Pixel)
go func() {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
a.mu.Lock()
for point, swatch := range a.grid {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
pipe <- Pixel{
X: point.X,
Y: point.Y,
Swatch: swatch,
}
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
a.mu.Unlock()
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
close(pipe)
}()
return pipe
}
// Get a pixel from the map.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) Get(p render.Point) (*Swatch, error) {
a.mu.Lock()
defer a.mu.Unlock()
pixel, ok := a.grid[p] // <- concurrent read and write
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
if !ok {
return nil, errors.New("no pixel")
}
return pixel, nil
}
// Set a pixel on the map.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) Set(p render.Point, sw *Swatch) error {
a.mu.Lock()
defer a.mu.Unlock()
a.grid[p] = sw
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
return nil
}
// Delete a pixel from the map.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) Delete(p render.Point) error {
a.mu.Lock()
defer a.mu.Unlock()
if _, ok := a.grid[p]; ok {
delete(a.grid, p)
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
return nil
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
return errors.New("pixel was not there")
}
// MarshalJSON to convert the chunk map to JSON.
//
// When serialized, the key is the "X,Y" coordinate and the value is the
// swatch index of the Palette, rather than redundantly serializing out the
// Swatch object for every pixel.
2023-02-18 20:45:36 +00:00
//
// DEPRECATED: in the Zipfile format chunks will be saved as binary files
// instead of with their JSON wrappers, so MarshalJSON will be phased out.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) MarshalJSON() ([]byte, error) {
a.mu.Lock()
defer a.mu.Unlock()
2023-02-18 20:45:36 +00:00
// Write in the new compressed format.
if balance.CompressMapAccessor {
var compressed []byte
for point, sw := range a.grid {
var (
x = int64(point.X)
y = int64(point.Y)
sw = uint64(sw.index)
entry = []byte{}
)
entry = binary.AppendVarint(entry, x)
entry = binary.AppendVarint(entry, y)
entry = binary.AppendUvarint(entry, sw)
compressed = append(compressed, entry...)
}
out, err := json.Marshal(compressed)
return out, err
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
dict := map[string]int{}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for point, sw := range a.grid {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
dict[point.String()] = sw.Index()
}
out, err := json.Marshal(dict)
return out, err
}
// UnmarshalJSON to convert the chunk map back from JSON.
2023-02-18 20:45:36 +00:00
//
// DEPRECATED: in the Zipfile format chunks will be saved as binary files
// instead of with their JSON wrappers, so MarshalJSON will be phased out.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
func (a *MapAccessor) UnmarshalJSON(b []byte) error {
a.mu.Lock()
defer a.mu.Unlock()
2023-02-18 20:45:36 +00:00
// Transparently upgrade the compression algorithm for this level.
// - Old style was a map[string]int like {"123,456": 4} mapping
// a coordinate to a palette index.
// - Now, coords and palettes are uint8 constrained so we can
// really tighten this up.
// For transparent upgrade, try and parse it the old way first.
var (
dict map[string]int // old-style
compressed []byte // new-style
)
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
err := json.Unmarshal(b, &dict)
if err != nil {
2023-02-18 20:45:36 +00:00
// Now try the new way.
err = json.Unmarshal(b, &compressed)
if err != nil {
return err
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
2023-02-18 20:45:36 +00:00
// New format: decompress the byte stream.
if compressed != nil {
// log.Debug("MapAccessor.Unmarshal: Reading %d bytes of compressed chunk data", len(compressed))
var (
reader = bytes.NewBuffer(compressed)
)
for {
var (
x, err1 = binary.ReadVarint(reader)
y, err2 = binary.ReadVarint(reader)
sw, err3 = binary.ReadUvarint(reader)
)
point := render.NewPoint(int(x), int(y))
a.grid[point] = NewSparseSwatch(int(sw))
if err1 != nil || err2 != nil || err3 != nil {
// log.Error("Break read loop: %s; %s; %s", err1, err2, err3)
break
}
}
return nil
}
// Old format: read the dict in.
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
for coord, index := range dict {
point, err := render.ParsePoint(coord)
if err != nil {
return fmt.Errorf("MapAccessor.UnmarshalJSON: %s", err)
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
a.grid[point] = NewSparseSwatch(index)
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
return nil
}
2023-02-18 20:45:36 +00:00
/*
MarshalBinary converts the chunk data to a binary representation, for
better compression compared to JSON.
In the binary format each chunk begins with one Varint (the chunk Type)
followed by whatever wire format the chunk needs given its type.
This function is related to the CompressMapAccessor config constant:
the MapAccessor compression boils down each point to a series if packed
varints: the X, Y coord (varint) followed by palette index (Uvarint).
The output of this function is just the compressed MapAccessor stream.
*/
func (a *MapAccessor) MarshalBinary() ([]byte, error) {
a.mu.Lock()
defer a.mu.Unlock()
// Write in the new compressed format.
var compressed []byte
for point, sw := range a.grid {
var (
x = int64(point.X)
y = int64(point.Y)
sw = uint64(sw.index)
entry = []byte{}
)
entry = binary.AppendVarint(entry, x)
entry = binary.AppendVarint(entry, y)
entry = binary.AppendUvarint(entry, sw)
compressed = append(compressed, entry...)
}
return compressed, nil
}
// UnmarshalBinary will decode a compressed MapAccessor byte stream.
func (a *MapAccessor) UnmarshalBinary(compressed []byte) error {
a.mu.Lock()
defer a.mu.Unlock()
// New format: decompress the byte stream.
2024-05-24 23:43:11 +00:00
// log.Debug("MapAccessor.Unmarshal: Reading %d bytes of compressed chunk data", len(compressed))
2023-02-18 20:45:36 +00:00
var reader = bytes.NewBuffer(compressed)
for {
var (
x, err1 = binary.ReadVarint(reader)
y, err2 = binary.ReadVarint(reader)
sw, err3 = binary.ReadUvarint(reader)
)
// We expect all 3 errors to be EOF together if the binary is formed correctly.
2023-02-18 20:45:36 +00:00
if err1 != nil || err2 != nil || err3 != nil {
if err1 == nil || err2 == nil || err3 == nil {
log.Error("MapAccessor.UnmarshalBinary: found odd number of varints!")
}
2023-02-18 20:45:36 +00:00
break
}
point := render.NewPoint(int(x), int(y))
a.grid[point] = NewSparseSwatch(int(sw))
2023-02-18 20:45:36 +00:00
}
return nil
}