doodle/pkg/doodads/fmt_zipfile.go

131 lines
2.7 KiB
Go
Raw Permalink Normal View History

Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
package doodads
import (
"archive/zip"
"bytes"
"encoding/json"
"fmt"
2022-09-24 22:17:25 +00:00
"git.kirsle.net/SketchyMaze/doodle/pkg/log"
"git.kirsle.net/go/render"
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
)
// ToZipfile serializes the doodad into zipfile format.
func (d *Doodad) ToZipfile() ([]byte, error) {
fh := bytes.NewBuffer([]byte{})
zipper := zip.NewWriter(fh)
defer zipper.Close()
// Migrate the Chunker caches into the zipfile.
for _, layer := range d.Layers {
if err := layer.Chunker.MigrateZipfile(zipper); err != nil {
return nil, fmt.Errorf("MigrateZipfile: %s", err)
}
}
// Migrate attached files to ZIP.
if d.Files != nil {
if err := d.Files.MigrateZipfile(zipper); err != nil {
return nil, fmt.Errorf("FileSystem.MigrateZipfile: %s", err)
}
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
// Write the header json.
{
header, err := d.AsJSON()
if err != nil {
return nil, err
}
writer, err := zipper.Create("doodad.json")
if err != nil {
return nil, err
}
if n, err := writer.Write(header); err != nil {
return nil, err
} else {
log.Debug("Written doodad.json to zipfile: %d bytes", n)
}
}
zipper.Close()
// Refresh our Zipfile reader from the zipper we just wrote.
bin := fh.Bytes()
if err := d.ReloadZipfile(bin); err != nil {
log.Error("ReloadZipfile: %s", err)
}
return fh.Bytes(), nil
}
// FromZipfile reads a doodad from zipfile format.
func FromZipfile(data []byte) (*Doodad, error) {
var (
doodad = New(0)
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
err = doodad.populateFromZipfile(data)
)
return doodad, err
}
// ReloadZipfile re-reads the level's zipfile after a write.
func (d *Doodad) ReloadZipfile(data []byte) error {
return d.populateFromZipfile(data)
}
// Common function between FromZipfile and ReloadZipFile.
func (d *Doodad) populateFromZipfile(data []byte) error {
var (
buf = bytes.NewReader(data)
zf *zip.Reader
decoder *json.Decoder
)
zf, err := zip.NewReader(buf, buf.Size())
if err != nil {
return err
}
// Read the doodad.json.
file, err := zf.Open("doodad.json")
if err != nil {
return err
}
decoder = json.NewDecoder(file)
err = decoder.Decode(d)
// Keep the zipfile reader handy.
d.Zipfile = zf
if d.Files != nil {
d.Files.Zipfile = zf
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
for i, layer := range d.Layers {
layer.Chunker.Layer = i
layer.Chunker.Zipfile = zf
}
// Re-inflate data after saving a new zipfile.
d.Inflate()
// If we are a legacy doodad and don't have a Size (width x height),
// set it from the chunk size.
if d.Size.IsZero() {
var size = d.ChunkSize()
d.Size = render.NewRect(size, size)
}
Zipfiles as File Format for Levels and Doodads Especially to further optimize memory for large levels, Levels and Doodads can now read and write to a ZIP file format on disk with chunks in external files within the zip. Existing doodads and levels can still load as normal, and will be converted into ZIP files on the next save: * The Chunker.ChunkMap which used to hold ALL chunks in the main json/gz file, now becomes the cache of "hot chunks" loaded from ZIP. If there is a ZIP file, chunks not accessed recently are flushed from the ChunkMap to save on memory. * During save, the ChunkMap is flushed to ZIP along with any non-loaded chunks from a previous zipfile. So legacy levels "just work" when saving, and levels loaded FROM Zip will manage their ChunkMap hot memory more carefully. Memory savings observed on "Azulian Tag - Forest.level": * Before: 1716 MB was loaded from the old level format into RAM along with a slow load screen. * After: only 243 MB memory was used by the game and it loaded with a VERY FAST load screen. Updates to the F3 Debug Overlay: * "Chunks: 20 in 45 out 20 cached" shows the count of chunks inside the viewport (having bitmaps and textures loaded) vs. chunks outside which have their textures freed (but data kept), and the number of chunks currently hot cached in the ChunkMap. The `doodad` tool has new commands to "touch" your existing levels and doodads, to upgrade them to the new format (or you can simply open and re-save them in-game): doodad edit-level --touch ./example.level doodad edit-doodad --touch ./example.doodad The output from that and `doodad show` should say "File format: zipfile" in the headers section. To do: * File attachments should also go in as ZIP files, e.g. wallpapers
2022-04-30 03:34:59 +00:00
return err
}
// Loop may be called each loop to allow the level to maintain its
// memory usage, e.g., for chunks not requested recently from a zipfile
// level to free those from RAM.
func (d *Doodad) Loop() error {
for _, layer := range d.Layers {
layer.Chunker.FreeCaches()
}
return nil
}