doodle/pkg/level/chunker.go

616 lines
16 KiB
Go
Raw Normal View History

Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
package level
import (
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
"archive/zip"
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
"encoding/json"
"fmt"
"math"
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
"sync"
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
"git.kirsle.net/apps/doodle/pkg/balance"
"git.kirsle.net/apps/doodle/pkg/log"
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
"git.kirsle.net/apps/doodle/pkg/shmem"
"git.kirsle.net/go/render"
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
)
// Chunker is the data structure that manages the chunks of a level, and
// provides the API to interact with the pixels using their absolute coordinates
// while abstracting away the underlying details.
type Chunker struct {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
// Layer is optional for the caller, levels use only 0 and
// doodads use them for frames. When chunks are exported to
// zipfile the Layer keeps them from overlapping.
Layer int `json:"-"` // internal use only
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
Size int `json:"size"`
// A Zipfile reference for new-style levels and doodads which
// keep their chunks in external parts of a zip file.
Zipfile *zip.Reader `json:"-"`
// Chunks, oh boy.
// The v1 drawing format had all the chunks in the JSON file.
// New drawings write them to zips. Legacy drawings can be converted
// simply by loading and resaving: their Chunks loads from JSON and
// is committed to zipfile on save. This makes Chunks also a good
// cache even when we have a zipfile to fall back on.
Chunks ChunkMap `json:"chunks"`
chunkMu sync.RWMutex
// If we have a zipfile, only keep chunks warm in memory if they
// are actively wanted by the game.
lastTick uint64 // NOTE: tracks from shmem.Tick
chunkRequestsThisTick map[render.Point]interface{}
requestsN1 map[render.Point]interface{} // chunks accessed last tick
requestsN2 map[render.Point]interface{} // 2 ticks ago (to free soon)
chunksToFree map[render.Point]uint64 // chopping block (free after X ticks)
ctfMu sync.Mutex // lock for chunksToFree
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
requestMu sync.Mutex
// The palette reference from first call to Inflate()
pal *Palette
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
// NewChunker creates a new chunk manager with a given chunk size.
func NewChunker(size int) *Chunker {
return &Chunker{
Size: size,
Chunks: ChunkMap{},
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
chunkRequestsThisTick: map[render.Point]interface{}{},
requestsN1: map[render.Point]interface{}{},
requestsN2: map[render.Point]interface{}{},
chunksToFree: map[render.Point]uint64{},
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
}
// Inflate iterates over the pixels in the (loaded) chunks and expands any
// Sparse Swatches (which have only their palette index, from the file format
// on disk) to connect references to the swatches in the palette.
func (c *Chunker) Inflate(pal *Palette) error {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
c.pal = pal
c.chunkMu.RLock()
defer c.chunkMu.RUnlock()
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
for coord, chunk := range c.Chunks {
WIP Texture Caching NOTICE: Chunk size set to 100 for visual testing! NOTICE: guitest references a bmp file that isn't checked in! BUGS REMAINING: - When scrolling the level in Edit Mode, some of the chunks will pop out of existence randomly. - When clicking-dragging to draw in Edit Mode, if the scroll position is not at 0,0 then the pixels drawn will be offset from the cursor. - These are to do with the Scroll position and chunk coordinate calc functions probably. Implements a texture caching interface to stop redrawing everything pixel by pixel on every frame. The texture caching workflow is briefly: - The uix.Canvas widget's Present() function iterates over the list of Chunk Coordinates that are visible inside of the current viewport (i.e. viewable on screen) - For each Chunk: - Make it render and/or return its cached Texture object. - Work out how much of the Chunk will be visible and how to crop the boxes for the Copy() - Copy the cached Texture instead of drawing all the pixels every time like we were doing before. - The Chunk.Texture() function that returns said Texture: - It calls Chunk.ToBitmap() to save a bitmap on disk. - It calls Engine.NewBitmap() to get a Texture it can hang onto. - It hangs onto the Texture and returns it on future calls. - Any call to Set() or Delete() a pixel will invalidate the cache (mark the Chunk "dirty") and Texture() will rebuild next call. The interface `render.Texturer` provides a way for rendering backends (SDL2, OpenGL) to transport a "texture" of their own kind without exposing the type details to the user. The interface `render.Engine` adds two new methods: * NewBitmap(filename string) (Texturer, error) * Copy(t Texturer, src, dst Rect) NewBitmap should open a bitmap image on disk and return it wrapped in a Texturer (really it's an SDL2 Texture). This is for caching purposes. Next the Copy() function blits the texture onto the screen renderer using the source and destination rectangles. The uix.Canvas widget orchestrates the caching for the drawing it's responsible for. It queries which chunks are viewable in the Canvas viewport (scroll and bounding boxes), has each chunk render out their entire bitmap image to then cache them as SDL textures and then only _those_ need to be copied out to the renderer each frame. The frame rate now sits at a decent 60 FPS even when the drawing gets messy and full of lines. Each unique version of each chunk needs to render only one time and then it's a fast copy operation for future ticks. Other changes: - Chunker now assigns each Chunk what their coordinate and size are, so that the chunk can self reference that information. This info is considered read-only but that isn't really enforced. - Add Chunker.IterViewportChunks() that returns a channel of Chunk Coordinates that are visible in your viewport, rather than iterating over all of the pixels in all of those chunks. - Add Chunk.ToBitmap(filename) that causes a Chunk to render its pixels to a bitmap image on disk. SDL2 can natively speak Bitmaps for texture caching. Currently these go to files in /tmp but will soon go into your $XDG_CACHE_FOLDER instead. - Add Chunk.Texture() that causes a Chunk to render and then return a cached bitmap texture of the pixels it's responsible for. The texture is cached until the Chunk is next modified with Set() or Delete(). - UI: add an Image widget that currently just shows a bitmap image. It was the first test for caching bitmap images for efficiency. Can show any *.bmp file on disk! - Editor UI: make the StatusBar boxes dynamically build from an array of string pointers to make it SUPER EASY to add/remove labels.
2018-10-18 03:52:14 +00:00
chunk.Point = coord
chunk.Size = c.Size
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
chunk.Inflate(pal)
}
return nil
}
// IterViewport returns a channel to iterate every point that exists within
// the viewport rect.
func (c *Chunker) IterViewport(viewport render.Rect) <-chan Pixel {
pipe := make(chan Pixel)
go func() {
// Get the chunk box coordinates.
var (
topLeft = c.ChunkCoordinate(render.NewPoint(viewport.X, viewport.Y))
bottomRight = c.ChunkCoordinate(render.Point{
X: viewport.X + viewport.W,
Y: viewport.Y + viewport.H,
})
)
for cx := topLeft.X; cx <= bottomRight.X; cx++ {
for cy := topLeft.Y; cy <= bottomRight.Y; cy++ {
if chunk, ok := c.GetChunk(render.NewPoint(cx, cy)); ok {
for px := range chunk.Iter() {
// Verify this pixel is also in range.
if px.Point().Inside(viewport) {
pipe <- px
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
}
}
}
close(pipe)
}()
return pipe
}
Optimize memory by freeing up SDL2 textures * Added to the F3 Debug Overlay is a "Texture:" label that counts the number of textures currently loaded by the (SDL2) render engine. * Added Teardown() functions to Level, Doodad and the Chunker they both use to free up SDL2 textures for all their cached graphics. * The Canvas.Destroy() function now cleans up all textures that the Canvas is responsible for: calling the Teardown() of the Level or Doodad, calling Destroy() on all level actors, and cleaning up Wallpaper textures. * The Destroy() method of the game's various Scenes will properly Destroy() their canvases to clean up when transitioning to another scene. The MainScene, MenuScene, EditorScene and PlayScene. * Fix the sprites package to actually cache the ui.Image widgets. The game has very few sprites so no need to free them just yet. Some tricky places that were leaking textures have been cleaned up: * Canvas.InstallActors() destroys the canvases of existing actors before it reinitializes the list and installs the replacements. * The DraggableActor when the user is dragging an actor around their level cleans up the blueprint masked drag/drop actor before nulling it out. Misc changes: * The player character cheats during Play Mode will immediately swap out the player character on the current level. * Properly call the Close() function instead of Hide() to dismiss popup windows. The Close() function itself calls Hide() but also triggers WindowClose event handlers. The Doodad Dropper subscribes to its close event to free textures for all its doodad canvases.
2022-04-09 21:41:24 +00:00
// IterChunks returns a channel to iterate over all chunks in the drawing.
func (c *Chunker) IterChunks() <-chan render.Point {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
var (
pipe = make(chan render.Point)
sent = map[render.Point]interface{}{}
)
Optimize memory by freeing up SDL2 textures * Added to the F3 Debug Overlay is a "Texture:" label that counts the number of textures currently loaded by the (SDL2) render engine. * Added Teardown() functions to Level, Doodad and the Chunker they both use to free up SDL2 textures for all their cached graphics. * The Canvas.Destroy() function now cleans up all textures that the Canvas is responsible for: calling the Teardown() of the Level or Doodad, calling Destroy() on all level actors, and cleaning up Wallpaper textures. * The Destroy() method of the game's various Scenes will properly Destroy() their canvases to clean up when transitioning to another scene. The MainScene, MenuScene, EditorScene and PlayScene. * Fix the sprites package to actually cache the ui.Image widgets. The game has very few sprites so no need to free them just yet. Some tricky places that were leaking textures have been cleaned up: * Canvas.InstallActors() destroys the canvases of existing actors before it reinitializes the list and installs the replacements. * The DraggableActor when the user is dragging an actor around their level cleans up the blueprint masked drag/drop actor before nulling it out. Misc changes: * The player character cheats during Play Mode will immediately swap out the player character on the current level. * Properly call the Close() function instead of Hide() to dismiss popup windows. The Close() function itself calls Hide() but also triggers WindowClose event handlers. The Doodad Dropper subscribes to its close event to free textures for all its doodad canvases.
2022-04-09 21:41:24 +00:00
go func() {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
c.chunkMu.RLock()
// Send the chunk coords we have in working memory.
// v1 levels: had all their chunks there in their JSON data
// v2 levels: chunks are in zipfile, cached ones are here
Optimize memory by freeing up SDL2 textures * Added to the F3 Debug Overlay is a "Texture:" label that counts the number of textures currently loaded by the (SDL2) render engine. * Added Teardown() functions to Level, Doodad and the Chunker they both use to free up SDL2 textures for all their cached graphics. * The Canvas.Destroy() function now cleans up all textures that the Canvas is responsible for: calling the Teardown() of the Level or Doodad, calling Destroy() on all level actors, and cleaning up Wallpaper textures. * The Destroy() method of the game's various Scenes will properly Destroy() their canvases to clean up when transitioning to another scene. The MainScene, MenuScene, EditorScene and PlayScene. * Fix the sprites package to actually cache the ui.Image widgets. The game has very few sprites so no need to free them just yet. Some tricky places that were leaking textures have been cleaned up: * Canvas.InstallActors() destroys the canvases of existing actors before it reinitializes the list and installs the replacements. * The DraggableActor when the user is dragging an actor around their level cleans up the blueprint masked drag/drop actor before nulling it out. Misc changes: * The player character cheats during Play Mode will immediately swap out the player character on the current level. * Properly call the Close() function instead of Hide() to dismiss popup windows. The Close() function itself calls Hide() but also triggers WindowClose event handlers. The Doodad Dropper subscribes to its close event to free textures for all its doodad canvases.
2022-04-09 21:41:24 +00:00
for point := range c.Chunks {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
sent[point] = nil
Optimize memory by freeing up SDL2 textures * Added to the F3 Debug Overlay is a "Texture:" label that counts the number of textures currently loaded by the (SDL2) render engine. * Added Teardown() functions to Level, Doodad and the Chunker they both use to free up SDL2 textures for all their cached graphics. * The Canvas.Destroy() function now cleans up all textures that the Canvas is responsible for: calling the Teardown() of the Level or Doodad, calling Destroy() on all level actors, and cleaning up Wallpaper textures. * The Destroy() method of the game's various Scenes will properly Destroy() their canvases to clean up when transitioning to another scene. The MainScene, MenuScene, EditorScene and PlayScene. * Fix the sprites package to actually cache the ui.Image widgets. The game has very few sprites so no need to free them just yet. Some tricky places that were leaking textures have been cleaned up: * Canvas.InstallActors() destroys the canvases of existing actors before it reinitializes the list and installs the replacements. * The DraggableActor when the user is dragging an actor around their level cleans up the blueprint masked drag/drop actor before nulling it out. Misc changes: * The player character cheats during Play Mode will immediately swap out the player character on the current level. * Properly call the Close() function instead of Hide() to dismiss popup windows. The Close() function itself calls Hide() but also triggers WindowClose event handlers. The Doodad Dropper subscribes to its close event to free textures for all its doodad canvases.
2022-04-09 21:41:24 +00:00
pipe <- point
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
c.chunkMu.RUnlock()
// If we have a zipfile, send any remaining chunks that are
// in colder storage.
if c.Zipfile != nil {
for _, point := range ChunksInZipfile(c.Zipfile, c.Layer) {
if _, ok := sent[point]; ok {
continue // Already sent from active memory
}
pipe <- point
}
}
close(pipe)
}()
return pipe
}
/*
IterChunksThemselves iterates all chunks in the drawing rather than coords.
Note: this will mark every chunk as "touched" this frame, so in a zipfile
level will load ALL chunks into memory.
*/
func (c *Chunker) IterChunksThemselves() <-chan *Chunk {
pipe := make(chan *Chunk)
go func() {
for coord := range c.IterChunks() {
if chunk, ok := c.GetChunk(coord); ok {
pipe <- chunk
}
}
close(pipe)
}()
return pipe
}
// IterCachedChunks iterates ONLY over the chunks currently cached in memory,
// e.g. so they can be torn down without loading extra chunks by looping normally.
func (c *Chunker) IterCachedChunks() <-chan *Chunk {
pipe := make(chan *Chunk)
go func() {
c.chunkMu.RLock()
defer c.chunkMu.RUnlock()
for _, chunk := range c.Chunks {
pipe <- chunk
}
Optimize memory by freeing up SDL2 textures * Added to the F3 Debug Overlay is a "Texture:" label that counts the number of textures currently loaded by the (SDL2) render engine. * Added Teardown() functions to Level, Doodad and the Chunker they both use to free up SDL2 textures for all their cached graphics. * The Canvas.Destroy() function now cleans up all textures that the Canvas is responsible for: calling the Teardown() of the Level or Doodad, calling Destroy() on all level actors, and cleaning up Wallpaper textures. * The Destroy() method of the game's various Scenes will properly Destroy() their canvases to clean up when transitioning to another scene. The MainScene, MenuScene, EditorScene and PlayScene. * Fix the sprites package to actually cache the ui.Image widgets. The game has very few sprites so no need to free them just yet. Some tricky places that were leaking textures have been cleaned up: * Canvas.InstallActors() destroys the canvases of existing actors before it reinitializes the list and installs the replacements. * The DraggableActor when the user is dragging an actor around their level cleans up the blueprint masked drag/drop actor before nulling it out. Misc changes: * The player character cheats during Play Mode will immediately swap out the player character on the current level. * Properly call the Close() function instead of Hide() to dismiss popup windows. The Close() function itself calls Hide() but also triggers WindowClose event handlers. The Doodad Dropper subscribes to its close event to free textures for all its doodad canvases.
2022-04-09 21:41:24 +00:00
close(pipe)
}()
return pipe
}
WIP Texture Caching NOTICE: Chunk size set to 100 for visual testing! NOTICE: guitest references a bmp file that isn't checked in! BUGS REMAINING: - When scrolling the level in Edit Mode, some of the chunks will pop out of existence randomly. - When clicking-dragging to draw in Edit Mode, if the scroll position is not at 0,0 then the pixels drawn will be offset from the cursor. - These are to do with the Scroll position and chunk coordinate calc functions probably. Implements a texture caching interface to stop redrawing everything pixel by pixel on every frame. The texture caching workflow is briefly: - The uix.Canvas widget's Present() function iterates over the list of Chunk Coordinates that are visible inside of the current viewport (i.e. viewable on screen) - For each Chunk: - Make it render and/or return its cached Texture object. - Work out how much of the Chunk will be visible and how to crop the boxes for the Copy() - Copy the cached Texture instead of drawing all the pixels every time like we were doing before. - The Chunk.Texture() function that returns said Texture: - It calls Chunk.ToBitmap() to save a bitmap on disk. - It calls Engine.NewBitmap() to get a Texture it can hang onto. - It hangs onto the Texture and returns it on future calls. - Any call to Set() or Delete() a pixel will invalidate the cache (mark the Chunk "dirty") and Texture() will rebuild next call. The interface `render.Texturer` provides a way for rendering backends (SDL2, OpenGL) to transport a "texture" of their own kind without exposing the type details to the user. The interface `render.Engine` adds two new methods: * NewBitmap(filename string) (Texturer, error) * Copy(t Texturer, src, dst Rect) NewBitmap should open a bitmap image on disk and return it wrapped in a Texturer (really it's an SDL2 Texture). This is for caching purposes. Next the Copy() function blits the texture onto the screen renderer using the source and destination rectangles. The uix.Canvas widget orchestrates the caching for the drawing it's responsible for. It queries which chunks are viewable in the Canvas viewport (scroll and bounding boxes), has each chunk render out their entire bitmap image to then cache them as SDL textures and then only _those_ need to be copied out to the renderer each frame. The frame rate now sits at a decent 60 FPS even when the drawing gets messy and full of lines. Each unique version of each chunk needs to render only one time and then it's a fast copy operation for future ticks. Other changes: - Chunker now assigns each Chunk what their coordinate and size are, so that the chunk can self reference that information. This info is considered read-only but that isn't really enforced. - Add Chunker.IterViewportChunks() that returns a channel of Chunk Coordinates that are visible in your viewport, rather than iterating over all of the pixels in all of those chunks. - Add Chunk.ToBitmap(filename) that causes a Chunk to render its pixels to a bitmap image on disk. SDL2 can natively speak Bitmaps for texture caching. Currently these go to files in /tmp but will soon go into your $XDG_CACHE_FOLDER instead. - Add Chunk.Texture() that causes a Chunk to render and then return a cached bitmap texture of the pixels it's responsible for. The texture is cached until the Chunk is next modified with Set() or Delete(). - UI: add an Image widget that currently just shows a bitmap image. It was the first test for caching bitmap images for efficiency. Can show any *.bmp file on disk! - Editor UI: make the StatusBar boxes dynamically build from an array of string pointers to make it SUPER EASY to add/remove labels.
2018-10-18 03:52:14 +00:00
// IterViewportChunks returns a channel to iterate over the Chunk objects that
// appear within the viewport rect, instead of the pixels in each chunk.
func (c *Chunker) IterViewportChunks(viewport render.Rect) <-chan render.Point {
pipe := make(chan render.Point)
go func() {
sent := make(map[render.Point]interface{})
for x := viewport.X; x < viewport.W; x += (c.Size / 4) {
for y := viewport.Y; y < viewport.H; y += (c.Size / 4) {
WIP Texture Caching NOTICE: Chunk size set to 100 for visual testing! NOTICE: guitest references a bmp file that isn't checked in! BUGS REMAINING: - When scrolling the level in Edit Mode, some of the chunks will pop out of existence randomly. - When clicking-dragging to draw in Edit Mode, if the scroll position is not at 0,0 then the pixels drawn will be offset from the cursor. - These are to do with the Scroll position and chunk coordinate calc functions probably. Implements a texture caching interface to stop redrawing everything pixel by pixel on every frame. The texture caching workflow is briefly: - The uix.Canvas widget's Present() function iterates over the list of Chunk Coordinates that are visible inside of the current viewport (i.e. viewable on screen) - For each Chunk: - Make it render and/or return its cached Texture object. - Work out how much of the Chunk will be visible and how to crop the boxes for the Copy() - Copy the cached Texture instead of drawing all the pixels every time like we were doing before. - The Chunk.Texture() function that returns said Texture: - It calls Chunk.ToBitmap() to save a bitmap on disk. - It calls Engine.NewBitmap() to get a Texture it can hang onto. - It hangs onto the Texture and returns it on future calls. - Any call to Set() or Delete() a pixel will invalidate the cache (mark the Chunk "dirty") and Texture() will rebuild next call. The interface `render.Texturer` provides a way for rendering backends (SDL2, OpenGL) to transport a "texture" of their own kind without exposing the type details to the user. The interface `render.Engine` adds two new methods: * NewBitmap(filename string) (Texturer, error) * Copy(t Texturer, src, dst Rect) NewBitmap should open a bitmap image on disk and return it wrapped in a Texturer (really it's an SDL2 Texture). This is for caching purposes. Next the Copy() function blits the texture onto the screen renderer using the source and destination rectangles. The uix.Canvas widget orchestrates the caching for the drawing it's responsible for. It queries which chunks are viewable in the Canvas viewport (scroll and bounding boxes), has each chunk render out their entire bitmap image to then cache them as SDL textures and then only _those_ need to be copied out to the renderer each frame. The frame rate now sits at a decent 60 FPS even when the drawing gets messy and full of lines. Each unique version of each chunk needs to render only one time and then it's a fast copy operation for future ticks. Other changes: - Chunker now assigns each Chunk what their coordinate and size are, so that the chunk can self reference that information. This info is considered read-only but that isn't really enforced. - Add Chunker.IterViewportChunks() that returns a channel of Chunk Coordinates that are visible in your viewport, rather than iterating over all of the pixels in all of those chunks. - Add Chunk.ToBitmap(filename) that causes a Chunk to render its pixels to a bitmap image on disk. SDL2 can natively speak Bitmaps for texture caching. Currently these go to files in /tmp but will soon go into your $XDG_CACHE_FOLDER instead. - Add Chunk.Texture() that causes a Chunk to render and then return a cached bitmap texture of the pixels it's responsible for. The texture is cached until the Chunk is next modified with Set() or Delete(). - UI: add an Image widget that currently just shows a bitmap image. It was the first test for caching bitmap images for efficiency. Can show any *.bmp file on disk! - Editor UI: make the StatusBar boxes dynamically build from an array of string pointers to make it SUPER EASY to add/remove labels.
2018-10-18 03:52:14 +00:00
// Constrain this chunksize step to a point within the bounds
// of the viewport. This can yield partial chunks on the edges
// of the viewport.
point := render.NewPoint(x, y)
if point.X < viewport.X {
point.X = viewport.X
} else if point.X > viewport.X+viewport.W {
point.X = viewport.X + viewport.W
}
if point.Y < viewport.Y {
point.Y = viewport.Y
} else if point.Y > viewport.Y+viewport.H {
point.Y = viewport.Y + viewport.H
}
// Translate to a chunk coordinate, dedupe and send it.
coord := c.ChunkCoordinate(render.NewPoint(x, y))
if _, ok := sent[coord]; ok {
continue
}
sent[coord] = nil
if _, ok := c.GetChunk(coord); ok {
pipe <- coord
}
}
}
close(pipe)
}()
return pipe
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
// IterPixels returns a channel to iterate over every pixel in the entire
// chunker.
func (c *Chunker) IterPixels() <-chan Pixel {
pipe := make(chan Pixel)
go func() {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for chunk := range c.IterChunksThemselves() {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
for px := range chunk.Iter() {
pipe <- px
}
}
close(pipe)
}()
return pipe
}
// WorldSize returns the bounding coordinates that the Chunker has chunks to
// manage: the lowest pixels from the lowest chunks to the highest pixels of
// the highest chunks.
func (c *Chunker) WorldSize() render.Rect {
chunkLowest, chunkHighest := c.Bounds()
return render.Rect{
X: chunkLowest.X * c.Size,
Y: chunkLowest.Y * c.Size,
W: (chunkHighest.X * c.Size) + (c.Size - 1),
H: (chunkHighest.Y * c.Size) + (c.Size - 1),
}
}
// WorldSizePositive returns the WorldSize anchored to 0,0 with only positive
// coordinates.
func (c *Chunker) WorldSizePositive() render.Rect {
S := c.WorldSize()
return render.Rect{
X: 0,
Y: 0,
W: int(math.Abs(float64(S.X))) + S.W,
H: int(math.Abs(float64(S.Y))) + S.H,
}
}
// Bounds returns the boundary points of the lowest and highest chunk which
// have any data in them.
func (c *Chunker) Bounds() (low, high render.Point) {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for coord := range c.IterChunks() {
if coord.X < low.X {
low.X = coord.X
}
if coord.Y < low.Y {
low.Y = coord.Y
}
if coord.X > high.X {
high.X = coord.X
}
if coord.Y > high.Y {
high.Y = coord.Y
}
}
return low, high
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
/*
GetChunk gets a chunk at a certain position. Returns false if not found.
This should be the centralized function to request a Chunk from the Chunker
(or IterChunksThemselves). On old-style levels all of the chunks were just
in memory as part of the JSON struct, in Zip files we can load/unload them
at will from external files.
*/
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
func (c *Chunker) GetChunk(p render.Point) (*Chunk, bool) {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
// It's currently cached in memory?
c.chunkMu.RLock()
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
chunk, ok := c.Chunks[p]
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
c.chunkMu.RUnlock()
// Was it on the chopping block for garbage collection?
c.ctfMu.Lock()
delete(c.chunksToFree, p)
c.ctfMu.Unlock()
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
if ok {
// An empty chunk? We hang onto these until save time to commit
// the empty chunk to ZIP.
if chunk.Len() == 0 {
return nil, false
}
c.logChunkAccess(p, chunk) // for the LRU cache
return chunk, ok
}
// Hit the zipfile for it.
if c.Zipfile != nil {
if chunk, err := ChunkFromZipfile(c.Zipfile, c.Layer, p); err == nil {
// log.Debug("GetChunk(%s) cache miss, read from zip", p)
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
c.SetChunk(p, chunk) // cache it
c.logChunkAccess(p, chunk) // for the LRU cache
if c.pal != nil {
chunk.Point = p
chunk.Size = c.Size
chunk.Inflate(c.pal)
}
return chunk, true
}
}
// Is our chunk cache getting too full? e.g. on full level
// sweeps where a whole zip file's worth of chunks are scanned.
if balance.ChunkerLRUCacheMax > 0 && len(c.Chunks) > balance.ChunkerLRUCacheMax {
log.Error("Chunks in memory (%d) exceeds LRU cache cap of %d, freeing random chunks")
c.chunkMu.Lock()
defer c.chunkMu.Unlock()
var (
i = 0
limit = len(c.Chunks) - balance.ChunkerLRUCacheMax
)
for coord := range c.Chunks {
if i < limit {
delete(c.Chunks, coord)
}
i++
}
}
return nil, false
}
// LRU cache for chunks from zipfiles: log which chunks were accessed
// this tick, so they can be compared to the tick prior, and then freed
// up after that.
func (c *Chunker) logChunkAccess(p render.Point, chunk *Chunk) {
// Record this point.
c.requestMu.Lock()
if c.chunkRequestsThisTick == nil {
c.chunkRequestsThisTick = map[render.Point]interface{}{}
}
c.chunkRequestsThisTick[p] = nil
c.requestMu.Unlock()
}
// FreeCaches unloads chunks that have not been requested in 2 frames.
//
// Only on chunkers that have zipfiles, old-style levels without zips
// wouldn't be able to restore their chunks otherwise! Returns -1 if
// no Zipfile, otherwise number of chunks freed.
func (c *Chunker) FreeCaches() int {
if c.Zipfile == nil {
return -1
}
var thisTick = shmem.Tick
// Very first tick this chunker has seen?
if c.lastTick == 0 {
c.lastTick = thisTick
}
// A new tick?
if (thisTick-c.lastTick)%4 == 0 {
c.requestMu.Lock()
c.chunkMu.Lock()
defer c.requestMu.Unlock()
defer c.chunkMu.Unlock()
var (
requestsThisTick = c.chunkRequestsThisTick
requestsN2 = c.requestsN2
delete_coords = []render.Point{}
)
// Chunks requested 2 ticks ago but not this tick, put on the chopping
// block to free them later.
c.ctfMu.Lock()
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for coord := range requestsN2 {
// Old point not requested recently?
if _, ok := requestsThisTick[coord]; !ok {
c.chunksToFree[coord] = shmem.Tick + balance.CanvasChunkFreeChoppingBlockTicks
}
}
// From the chopping block, see if scheduled chunks to free are ready.
for coord, expireAt := range c.chunksToFree {
if shmem.Tick > expireAt {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
delete_coords = append(delete_coords, coord)
}
}
// Free any eligible chunks NOW.
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for _, coord := range delete_coords {
delete(c.chunksToFree, coord)
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
c.FreeChunk(coord)
}
c.ctfMu.Unlock()
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
// Rotate the cached ticks and clean the slate.
c.requestsN2 = c.requestsN1
c.requestsN1 = requestsThisTick
c.chunkRequestsThisTick = map[render.Point]interface{}{}
c.lastTick = thisTick
return len(delete_coords)
}
return 0
}
// SetChunk writes the chunk into the cache dict and nothing more.
//
// This function should be the singular writer to the chunk cache.
func (c *Chunker) SetChunk(p render.Point, chunk *Chunk) {
c.chunkMu.Lock()
c.Chunks[p] = chunk
c.chunkMu.Unlock()
c.logChunkAccess(p, chunk)
}
// FreeChunk unloads a chunk from active memory for zipfile-backed levels.
//
// Not thread safe: it is assumed the caller has the lock on c.Chunks.
func (c *Chunker) FreeChunk(p render.Point) bool {
if c.Zipfile == nil {
return false
}
// If this chunk has been modified since it was last loaded from ZIP, hang onto it
// in memory until the next save so we don't lose it.
if chunk, ok := c.Chunks[p]; ok {
if chunk.IsModified() {
return false
}
// Don't delete empty chunks, hang on until next zipfile save.
if chunk, ok := c.Chunks[p]; ok && chunk.Len() == 0 {
return false
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
}
delete(c.Chunks, p)
return true
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
// Redraw marks every chunk as dirty and invalidates all their texture caches,
// forcing the drawing to re-generate from scratch.
func (c *Chunker) Redraw() {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for chunk := range c.IterChunksThemselves() {
chunk.SetDirty()
}
}
// Prerender visits every chunk and fetches its texture, in order to pre-load
// the whole drawing for smooth gameplay rather than chunks lazy rendering as
// they enter the screen.
func (c *Chunker) Prerender() {
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for chunk := range c.IterChunksThemselves() {
_ = chunk.CachedBitmap(render.Invisible)
}
}
// PrerenderN will pre-render the texture for N number of chunks and then
// yield back to the caller. Returns the number of chunks that still need
// textures rendered; zero when the last chunk has been prerendered.
func (c *Chunker) PrerenderN(n int) (remaining int) {
var (
total int // total no. of chunks available
totalRendered int // no. of chunks with textures
modified int // number modified this call
)
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for chunk := range c.IterChunksThemselves() {
total++
if chunk.bitmap != nil {
totalRendered++
continue
}
if modified < n {
_ = chunk.CachedBitmap(render.Invisible)
totalRendered++
modified++
}
}
remaining = total - totalRendered
return
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
// Get a pixel at the given coordinate. Returns the Palette entry for that
// pixel or else returns an error if not found.
func (c *Chunker) Get(p render.Point) (*Swatch, error) {
// Compute the chunk coordinate.
coord := c.ChunkCoordinate(p)
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
if chunk, ok := c.GetChunk(coord); ok {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
return chunk.Get(p)
}
return nil, fmt.Errorf("no chunk %s exists for point %s", coord, p)
}
// Set a pixel at the given coordinate.
func (c *Chunker) Set(p render.Point, sw *Swatch) error {
coord := c.ChunkCoordinate(p)
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
chunk, ok := c.GetChunk(coord)
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
if !ok {
chunk = NewChunk()
WIP Texture Caching NOTICE: Chunk size set to 100 for visual testing! NOTICE: guitest references a bmp file that isn't checked in! BUGS REMAINING: - When scrolling the level in Edit Mode, some of the chunks will pop out of existence randomly. - When clicking-dragging to draw in Edit Mode, if the scroll position is not at 0,0 then the pixels drawn will be offset from the cursor. - These are to do with the Scroll position and chunk coordinate calc functions probably. Implements a texture caching interface to stop redrawing everything pixel by pixel on every frame. The texture caching workflow is briefly: - The uix.Canvas widget's Present() function iterates over the list of Chunk Coordinates that are visible inside of the current viewport (i.e. viewable on screen) - For each Chunk: - Make it render and/or return its cached Texture object. - Work out how much of the Chunk will be visible and how to crop the boxes for the Copy() - Copy the cached Texture instead of drawing all the pixels every time like we were doing before. - The Chunk.Texture() function that returns said Texture: - It calls Chunk.ToBitmap() to save a bitmap on disk. - It calls Engine.NewBitmap() to get a Texture it can hang onto. - It hangs onto the Texture and returns it on future calls. - Any call to Set() or Delete() a pixel will invalidate the cache (mark the Chunk "dirty") and Texture() will rebuild next call. The interface `render.Texturer` provides a way for rendering backends (SDL2, OpenGL) to transport a "texture" of their own kind without exposing the type details to the user. The interface `render.Engine` adds two new methods: * NewBitmap(filename string) (Texturer, error) * Copy(t Texturer, src, dst Rect) NewBitmap should open a bitmap image on disk and return it wrapped in a Texturer (really it's an SDL2 Texture). This is for caching purposes. Next the Copy() function blits the texture onto the screen renderer using the source and destination rectangles. The uix.Canvas widget orchestrates the caching for the drawing it's responsible for. It queries which chunks are viewable in the Canvas viewport (scroll and bounding boxes), has each chunk render out their entire bitmap image to then cache them as SDL textures and then only _those_ need to be copied out to the renderer each frame. The frame rate now sits at a decent 60 FPS even when the drawing gets messy and full of lines. Each unique version of each chunk needs to render only one time and then it's a fast copy operation for future ticks. Other changes: - Chunker now assigns each Chunk what their coordinate and size are, so that the chunk can self reference that information. This info is considered read-only but that isn't really enforced. - Add Chunker.IterViewportChunks() that returns a channel of Chunk Coordinates that are visible in your viewport, rather than iterating over all of the pixels in all of those chunks. - Add Chunk.ToBitmap(filename) that causes a Chunk to render its pixels to a bitmap image on disk. SDL2 can natively speak Bitmaps for texture caching. Currently these go to files in /tmp but will soon go into your $XDG_CACHE_FOLDER instead. - Add Chunk.Texture() that causes a Chunk to render and then return a cached bitmap texture of the pixels it's responsible for. The texture is cached until the Chunk is next modified with Set() or Delete(). - UI: add an Image widget that currently just shows a bitmap image. It was the first test for caching bitmap images for efficiency. Can show any *.bmp file on disk! - Editor UI: make the StatusBar boxes dynamically build from an array of string pointers to make it SUPER EASY to add/remove labels.
2018-10-18 03:52:14 +00:00
chunk.Point = coord
chunk.Size = c.Size
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
c.SetChunk(coord, chunk)
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
}
return chunk.Set(p, sw)
}
// SetRect sets a rectangle of pixels to a color all at once.
func (c *Chunker) SetRect(r render.Rect, sw *Swatch) error {
var (
xMin = r.X
yMin = r.Y
xMax = r.X + r.W
yMax = r.Y + r.H
)
for x := xMin; x < xMax; x++ {
for y := yMin; y < yMax; y++ {
c.Set(render.NewPoint(x, y), sw)
}
}
return nil
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
// Delete a pixel at the given coordinate.
func (c *Chunker) Delete(p render.Point) error {
coord := c.ChunkCoordinate(p)
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
if chunk, ok := c.GetChunk(coord); ok {
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
return chunk.Delete(p)
}
return fmt.Errorf("no chunk %s exists for point %s", coord, p)
}
// DeleteRect deletes a rectangle of pixels between two points.
// The rect is a relative one with a width and height, and the X,Y values are
// an absolute world coordinate.
func (c *Chunker) DeleteRect(r render.Rect) error {
var (
xMin = r.X
yMin = r.Y
xMax = r.X + r.W
yMax = r.Y + r.H
)
for x := xMin; x < xMax; x++ {
for y := yMin; y < yMax; y++ {
c.Delete(render.NewPoint(x, y))
}
}
return nil
}
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
// ChunkCoordinate computes a chunk coordinate from an absolute coordinate.
func (c *Chunker) ChunkCoordinate(abs render.Point) render.Point {
if c.Size == 0 {
return render.Point{}
}
size := float64(c.Size)
return render.NewPoint(
int(math.Floor(float64(abs.X)/size)),
int(math.Floor(float64(abs.Y)/size)),
Implement Chunk System for Pixel Data Starts the implementation of the chunk-based pixel storage system for levels and drawings. Previously the levels had a Pixels structure which was just an array of X,Y and palette index triplets. The new chunk system divides the map up into square chunks, and lets each chunk manage its own memory layout. The "MapAccessor" layout is implemented first which is a map of X,Y coordinates to their Swatches (pointer to an index of the palette). When serialized the MapAccessor maps the "X,Y": "index" similarly to the old Pixels array. The object hierarchy for the chunk system is like: * Chunker: the manager of the chunks who keeps track of the ChunkSize and a map of "chunk coordinates" to the chunk in charge of it. * Chunk: a part of the drawing ChunkSize length square. A chunk has a Type (of how it stores its data, 0 being a map[Point]Swatch and 1 being a [][]Swatch 2D array), and the chunk has an Accessor which implements the underlying type. * Accessor: an interface for a Chunk to provide access to its pixels. * MapAccessor: a "sparse map" of coordinates to their Swatches. * GridAccessor: TBD, will be a "dense" 2D grid of Swatches. The JSON files are loaded in two passes: 1. The chunks only load their swatch indexes from disk. 2. With the palette also loaded, the chunks are "inflated" and linked to their swatch pointers. Misc changes: * The `level.Canvas` UI widget switches from the old Grid data type to being able to directly use a `level.Chunker` * The Chunker is a shared data type between the on-disk level format and the actual renderer (level.Canvas), so saving the level is easy because you can just pull the Chunker out from the canvas. * ChunkSize is stored inside the level file and the default value is at balance/numbers.go: 1000
2018-09-23 22:20:45 +00:00
)
}
// ChunkMap maps a chunk coordinate to its chunk data.
type ChunkMap map[render.Point]*Chunk
// MarshalJSON to convert the chunk map to JSON. This is needed for writing so
// the JSON encoder knows how to serializes a `map[Point]*Chunk` but the inverse
// is not necessary to implement.
func (c ChunkMap) MarshalJSON() ([]byte, error) {
dict := map[string]*Chunk{}
for point, chunk := range c {
dict[point.String()] = chunk
}
out, err := json.Marshal(dict)
return out, err
}