GZip for Level Files #33

New Issue

kirsle · 2020-11-27T20:17:20Z

kirsle commented

2020-11-27 20:17:20 +00:00

Currently, level and doodad files are in JSON format and they could benefit by some simple GZip compression.

For example, the "full-color Mario 1-1 example level" weighs 168 MB in size but after gzip compressing it, it reduced all the way down to 13 MB! (93% reduction). In Go, the gzip reader should transparently allow compression and decompression in level files.

Doodle should check for a gzip header when reading to tell whether the file is compressed or not, so legacy files can still be opened.

(Ticket was formerly: "SQLite as an Application File Format"; rejected because SQLite would make cross-compiling difficult)

https://www.sqlite.org/appfileformat.html

Levels and Doodads are currently stored as plain .json files and there's room for improvement (compression, obfuscation) and eventually the need to embed custom files inside of levels (attach custom doodads, wallpapers, etc.)

Instead of just moving to a gzipped JSON format or something more custom, SQLite may be a good next version to move to. It could use a couple of SQL tables to organize its data:

headers: key/value pairs to hold title, author, game version, etc. metadata
attachments: key/value file store for custom wallpapers, doodads, etc.
chunks: hold the drawing data, x/y chunk coordinate + json blob of its chunker data; json could be gzip compressed
actors: structure to hold doodad instances placed in the level.

Doodle could be programmed to read legacy json files as normal by checking for the opening { character at the head of the file, or new SQLite files by checking for a SQLite header.

Currently, level and doodad files are in JSON format and they could benefit by some simple GZip compression. For example, the "full-color Mario 1-1 example level" weighs 168 MB in size but after gzip compressing it, it reduced all the way down to 13 MB! (93% reduction). In Go, the gzip reader should transparently allow compression and decompression in level files. Doodle should check for a gzip header when reading to tell whether the file is compressed or not, so legacy files can still be opened. ----- (Ticket was formerly: "SQLite as an Application File Format"; rejected because SQLite would make cross-compiling difficult) https://www.sqlite.org/appfileformat.html Levels and Doodads are currently stored as plain .json files and there's room for improvement (compression, obfuscation) and eventually the need to embed custom files inside of levels (attach custom doodads, wallpapers, etc.) Instead of just moving to a gzipped JSON format or something more custom, SQLite may be a good next version to move to. It could use a couple of SQL tables to organize its data: * headers: key/value pairs to hold title, author, game version, etc. metadata * attachments: key/value file store for custom wallpapers, doodads, etc. * chunks: hold the drawing data, x/y chunk coordinate + json blob of its chunker data; json could be gzip compressed * actors: structure to hold doodad instances placed in the level. Doodle could be programmed to read legacy json files as normal by checking for the opening `{` character at the head of the file, or new SQLite files by checking for a SQLite header.

kirsle added the

enhancement

label 2020-11-27 20:17:20 +00:00

kirsle changed title from ~~SQLite as an Application File Format~~ to GZip for Level Files

2020-12-29 04:59:34 +00:00

kirsle closed this issue

2021-07-18 20:40:47 +00:00

Sign in to join this conversation.