Zipfile for Levels and Doodads #80

New Issue

kirsle · 2022-04-27T02:02:37Z

kirsle commented

2022-04-27 02:02:37 +00:00

Currently: levels are gzipped JSON files of type level.Level and all drawn chunks of the level are in there, which for very large and colorful levels, is a problem on memory.

Levels could instead be made as a zipfile, like levelpacks do. Chunks can be stored in separate files of the archive and accessed randomly in a more memory efficient way.

Layout of the zipfile like:

/index.json (the Level or Doodad struct as normal, sans the chunkers)
/chunks/0/100,200.json (layers and chunks for Levels and Doodads; levels using layer 0)
/files/* (any attached assets like wallpapers and embedded doodads)

Implementation Hints

In either level.Base or level.Level structs:

Add a property that can hold a Zipfile, if nil then it is an old-style .level gzip-json file.
If added to Base then doodads may also be able to use zipfiles too, tho Level is the biggest concern.

level.New():

Level.Base.Version++ to version 2?
Initializes a Zipfile and will write in the new format (per a boolProp to configure it old-style or new-style?)

level.Chunker

The Chunker is the main thing being ousted from the main json file, so the feature may be close to here.

It's used in two ways:

type Level struct{
	Chunker *Chunker
}

type Doodad struct {
	Layers []struct{
		Chunker *Chunker
	}
}

Chunkers should be given the Zipfile reference and be able to smartly proxy its API calls to handle reading chunks from the zipfile contents.

type Chunker struct {
	Size   int      `json:"size"`
	Chunks ChunkMap `json:"chunks"` // deprecated
	Zip    *Zipfile `json:"-"`      // NEW
}

On save, if the ChunkMap is populated migrate it into the zipfile and clear the json out. Migrating a drawing then means open it the old way (ChunkMap loaded from json), and on save, populate the zipfile chunks instead.
IterChunks(), IterPixels(), Bounds() and anything that touches c.Chunks:
1. Backwards compat, iterate over the ChunkMap as currently
2. And then iterate over the zipfile chunks, for any new UNIQUE
  coordinate not sent on step 1.
GetChunk() is a good centralized function that reads in a chunk.
- Backwards compat, use the ChunkMap if it exists.
- Look in the zipfile and return it.

Collision

The level collision detection knows nothing about the Chunker and expects to get an accessible grid of pixel data.

Replacing it by an interface to proxy its access to pixel data the Zipfile chunker can work.

level.FromJSON() and ToJSON()

Levels: FromJSON() and ToJSON() are the current root functions for the level file format, already supporting gzip or plain JSON files. FromJSON() currently inspects the file header to see if it begins with:

{ = plain text JSON file.
0x1f8b = gzip compressed JSON file.
New: check for a zipfile header to parse zipped levels.

ToJSON() currently defers straight to ToGzip() if compressing drawings, and FromJSON() defers to FromGzip() when it detects the gzip header.

Add a ToZipfile() that ToJSON() will call instead when the zipfile feature is enabled which would:

If the level wasn't loaded with a Zipfile, create a new one:
- Visit the Chunker and copy any legacy ChunkMap chunks into zip members
- Clear out the Chunker.ChunkMap so we can write the Level/Doodad as /index.json in the zipfile (headers only)
Write to the .level/.doodad file as the Zipfile.

Add a FromZipfile() function to read that back from disk:

Loads the /index.json to populate the Level/Doodad struct.
Into the Level/Doodad struct attach the *Zipfile handle and be sure the Chunkers can access it.

As the game calls GetChunk() to pull up chunks of interest, read and parse them from the zip file.

Caching, loading/unloading and mobile doodads

Problem: actors on a level are always active without culling. If the game were to completely unload the chunks that a far away mob is in, they would fall into the void.

The Chunker doesn't know and can't know about Actors but should have a way to either mark chunks to keep active or have a Least Recently Used cache. Ideas:

Marking chunks as important:
- The uix.Canvas knows where all the actors are and which ones are set IsMobile. On its Loop() function it could tell the level Chunker which chunks it should not unload from the zipfile.
- Or in case the chunks are not loaded: which chunks to load now and their neighboring chunks.
- On level start, all Mobile doodads' locations would be marked important and the Chunker would load these from the zipfile and keep them accessible.
- As mobile actors move, update the important chunk coordinates.
- Have a "clear chunk caches" function which unloads all chunks except for the ones marked important + a radius of neighboring chunks or a viewport of sorts surrounding each mob.
- Non-mobile doodads don't need to keep their surroundings loaded.
LRU cache alternative:
- The Chunker just keeps a buffer of (say) last 50 accessed chunks that it stores in RAM so if you ask again it skips the zipfile.
- Naturally the level Viewport will ask for the chunks visible on screen.
- Putting "viewports" around mobile actors and just pinging their locations keeps their data warmed up.
- Viewporting could be done around the collision checks as each mob is moving in the level: load a viewport of chunks in a large enough radius that the mob traverses the level OK.
- In case of cache misses (too many actors loading too many chunks) the zipfile will be accessed each tick with a possible performance hit.
LRU cache by "accessed last frame?" instead of a fixed size buffer of e.g. 50 last reads.
- Three variables needed:
  - map[Point]Chunk cache that holds the warm chunk data.
  - map[Point]nil thisFrame map to record which chunks we accessed this tick.
  - map[Point]nil lastFrame map holds the chunks we accessed the previous tick.
- Tick 1:
  - all variables are empty.
  - a set of chunks are read from for the viewport and whatever the game is looking at in your level, populating the cache and thisFrame.
  - End of tick: thisFrame is copied into lastFrame and thisFrame is reset to empty to collect the next tick's accesses.
- Tick 2:
  - Say one new chunk is accessed and one chunk last tick is not (scrolled off screen)
  - thisFrame is empty and collects only the chunks accessed this frame, and cache adds the newly requested chunk.
  - End of tick: thisFrame and lastFrame compare to see which chunk from lastFrame was not pinged thisFrame: its cache entry is deleted. thisFrame is copied to lastFrame and thisFrame reset to empty.
- Pros: cache size is exactly as big as it needs to be and frees up within one frame as soon as the game is no longer interested in a chunk.

Currently: levels are gzipped JSON files of type level.Level and all drawn chunks of the level are in there, which for very large and colorful levels, is a problem on memory. Levels could instead be made as a zipfile, like levelpacks do. Chunks can be stored in separate files of the archive and accessed randomly in a more memory efficient way. Layout of the zipfile like: * /index.json (the Level or Doodad struct as normal, sans the chunkers) * /chunks/0/100,200.json (layers and chunks for Levels and Doodads; levels using layer 0) * /files/* (any attached assets like wallpapers and embedded doodads) # Implementation Hints In either level.Base or level.Level structs: * Add a property that can hold a Zipfile, if nil then it is an old-style .level gzip-json file. * If added to Base then doodads may also be able to use zipfiles too, tho Level is the biggest concern. level.New(): * Level.Base.Version++ to version 2? * Initializes a Zipfile and will write in the new format (per a boolProp to configure it old-style or new-style?) ## level.Chunker The Chunker is the main thing being ousted from the main json file, so the feature may be close to here. It's used in two ways: ```go type Level struct{ Chunker *Chunker } type Doodad struct { Layers []struct{ Chunker *Chunker } } ``` Chunkers should be given the Zipfile reference and be able to smartly proxy its API calls to handle reading chunks from the zipfile contents. ```go type Chunker struct { Size int `json:"size"` Chunks ChunkMap `json:"chunks"` // deprecated Zip *Zipfile `json:"-"` // NEW } ``` * On save, if the ChunkMap is populated migrate it into the zipfile and clear the json out. Migrating a drawing then means open it the old way (ChunkMap loaded from json), and on save, populate the zipfile chunks instead. * IterChunks(), IterPixels(), Bounds() and anything that touches c.Chunks: 1. Backwards compat, iterate over the ChunkMap as currently 2. And then iterate over the zipfile chunks, for any new UNIQUE coordinate not sent on step 1. * GetChunk() is a good centralized function that reads in a chunk. * Backwards compat, use the ChunkMap if it exists. * Look in the zipfile and return it. ## Collision The level collision detection knows nothing about the Chunker and expects to get an accessible grid of pixel data. Replacing it by an interface to proxy its access to pixel data the Zipfile chunker can work. ## level.FromJSON() and ToJSON() Levels: FromJSON() and ToJSON() are the current root functions for the level file format, already supporting gzip or plain JSON files. FromJSON() currently inspects the file header to see if it begins with: * `{` = plain text JSON file. * `0x1f8b` = gzip compressed JSON file. * **New:** check for a zipfile header to parse zipped levels. ToJSON() currently defers straight to ToGzip() if compressing drawings, and FromJSON() defers to FromGzip() when it detects the gzip header. Add a ToZipfile() that ToJSON() will call instead when the zipfile feature is enabled which would: * If the level wasn't loaded with a Zipfile, create a new one: * Visit the Chunker and copy any legacy ChunkMap chunks into zip members * Clear out the Chunker.ChunkMap so we can write the Level/Doodad as `/index.json` in the zipfile (headers only) * Write to the .level/.doodad file as the Zipfile. Add a FromZipfile() function to read that back from disk: * Loads the /index.json to populate the Level/Doodad struct. * Into the Level/Doodad struct attach the `*Zipfile` handle and be sure the Chunkers can access it. As the game calls GetChunk() to pull up chunks of interest, read and parse them from the zip file. ## Caching, loading/unloading and mobile doodads Problem: actors on a level are always active without culling. If the game were to completely unload the chunks that a far away mob is in, they would fall into the void. The Chunker doesn't know and can't know about Actors but should have a way to either mark chunks to keep active or have a Least Recently Used cache. Ideas: * Marking chunks as important: * The uix.Canvas knows where all the actors are and which ones are set IsMobile. On its Loop() function it could tell the level Chunker which chunks it should not unload from the zipfile. * Or in case the chunks are _not_ loaded: which chunks to load _now_ and their neighboring chunks. * On level start, all Mobile doodads' locations would be marked important and the Chunker would load these from the zipfile and keep them accessible. * As mobile actors move, update the important chunk coordinates. * Have a "clear chunk caches" function which unloads all chunks _except_ for the ones marked important + a radius of neighboring chunks or a viewport of sorts surrounding each mob. * Non-mobile doodads don't need to keep their surroundings loaded. * LRU cache alternative: * The Chunker just keeps a buffer of (say) last 50 accessed chunks that it stores in RAM so if you ask again it skips the zipfile. * Naturally the level Viewport will ask for the chunks visible on screen. * Putting "viewports" around mobile actors and just pinging their locations keeps their data warmed up. * Viewporting could be done around the collision checks as each mob is moving in the level: load a viewport of chunks in a large enough radius that the mob traverses the level OK. * In case of cache misses (too many actors loading too many chunks) the zipfile will be accessed each tick with a possible performance hit. * LRU cache by "accessed last frame?" instead of a fixed size buffer of e.g. 50 last reads. * Three variables needed: * map[Point]Chunk **cache** that holds the warm chunk data. * map[Point]nil **thisFrame** map to record which chunks we accessed this tick. * map[Point]nil **lastFrame** map holds the chunks we accessed the previous tick. * Tick 1: * all variables are empty. * a set of chunks are read from for the viewport and whatever the game is looking at in your level, populating the cache and thisFrame. * End of tick: thisFrame is copied into lastFrame and thisFrame is reset to empty to collect the next tick's accesses. * Tick 2: * Say one new chunk is accessed and one chunk last tick is not (scrolled off screen) * thisFrame is empty and collects only the chunks accessed this frame, and cache adds the newly requested chunk. * End of tick: thisFrame and lastFrame compare to see which chunk from lastFrame was _not_ pinged thisFrame: its cache entry is deleted. thisFrame is copied to lastFrame and thisFrame reset to empty. * Pros: cache size is exactly as big as it needs to be and frees up within one frame as soon as the game is no longer interested in a chunk.

kirsle added the

enhancement

label 2022-04-27 02:02:37 +00:00

Sign in to join this conversation.