Persistence
This page aims to give an overview of the serialization options Minetest provides.
Your choice for persistence depends on the data you want to persist:
- Structure & types of the data: How is the data structured? What data types occur?
- Size of the data, frequency of changes to the data: Can it fit into memory? How expensive can updates be?
- Granularity needed: How often must the data be persisted?
- Required queryability of the data: Are indices for fast lookups required?
- Required data access, objects the data is tied to: Items? Players?
(De-)Serialization
JSON
Minetest provides a JSON serializer based on jsoncpp
.
The following types are supported by JSON:
- booleans
- numbers
- strings
- tables (objects/arrays)
NOTE: Negative and positive infinity are represented by numbers which exceed the double number range by a lot (plus/minus 1e9999
).
WARNING: The empty table {}
(ambiguous: might be either {}
or []
) and nan
aren't supported by JSON and will be turned into null
.
Tables may either:
- Contain only positive integer keys (represented as array) or
- Contain only string keys (represented as object/dictionary) obviously, plenty of Lua tables aren't representable this way (boolean keys, mixed hash/list keys, fractional keys, inf keys)
minetest.write_json(data, [styled])
If the data is serializable as outlined above:: Returns a JSON string representing data
. Formats the JSON nicely to improve readability if styled
is truthy.
Else:: Returns nil
and one of the following errors if it fails:
Can't use indexes with fractional part in JSON
Lua key to convert to JSON is not a string or number
Can't mix array and object values in JSON
WARNING: Hash tables containing only positive integer keys - a table of minetest.hash_node_position
hashes for instance -
will be turned into an array with holes, that is, all nil
values from 1
to the maximum index will be filled with null
,
which means terrible performance and possibly very large output generated from very small input.
Do not allow direct access to minetest.write_json
as it can be trivially used to DoS your server.
TIP: Sparse hash maps, should always be converted to objects, using only string keys, for JSON. You might do this just for serialization.
WARNING: The JSON serializer is furthermore limited by its recursion depth: Only a recursion depth up to 16 is allowed. Very nested data structures can't be (de)serialized. Circular references will cause the JSON serializer to recurse until the maximum recursion depth is exceeded.
TIP: Use the json = assert(minetest.write_json(data))
idiom to not silently ignore errors when serializing.
minetest.parse_json(json, [nullvalue])
If json
is valid JSON:: Returns the Lua value represented by the JSON string json
, with null
values deserialized to nullvalue
(which defaults to nil
).
Else:: Returns nil
and logs an error.
TIP: If you must use JSON (to interact with a web interface, for instance), consider using your JSON library of choice;
Lua-only implementations such as lunajson
are available.
Lua
Minetest alternatively offers a serializer implemented in Lua, which serializes to Lua source code, available as minetest.serialize
and minetest.deserialize
. The following values are supported:
nil
- booleans
- numbers
- strings
- tables, including circular references
- functions: not recommended; uses
string.dump
internally- Deserialization of functions will error unless mod security is disabled
- Lua-implementation and platform-specific bytecode: Functions serialized by PUC Lua won't work on LuaJIT and vice versa; no cross-platform portability
- Upvalues or the context of the function (function environment) aren't preserved
- C functions which lack Lua bytecode (such as
math.sqrt
or most Minetest API functions) can't be (de)serialized at all
Userdata objects (like ItemStack
for example) aren't supported. Threads (coroutines) aren't supported either.
minetest.serialize(data)
Serializes data
, which may be a value of any of the above types.
Returns a Lua chunk in string
form that returns data
if executed.
Will error with Can't serialize data of type <type>
if it encounters an unsupported type.
Fully supports circular references.
minetest.deserialize(str, [safe])
If str
is not of type string
:: Returns nil, "Cannot deserialize type '<type>'. Argument must be a string."
If str
starts with "\27"
:: Returns nil, "Bytecode prohibited"
If str
triggers a Lua error at run- or load-time:: Returns nil, error
If str
returns without errors:: Returns the first value returned by executing the chunk str
without arguments
If safe
is truthy, serialized functions will be deserialized to nil
.
This will trigger an error if functions are used as table keys ({[function()end] = true}
).
Otherwise, serialized functions will get an empty function environment set - only being able to operate on literals and arguments.
TIP: Use of the data = assert(minetest.deserialize(lua, safe))
idiom is recommended.
WARNING: minetest.deserialize
errors on large objects on LuaJIT
Engine-provided default persistence
Nodes (consisting of nodename, param1, param2, meta) are persisted automatically as part of mapblocks.
A handful of player properties (HP, breath, position, look direction, inventory, meta) are persisted as well.
Granularity is controlled by the server_map_save_interval
setting.
Storage options
Database server / the ominous cloud
You can use Minetest's HTTP library to communicate with webservers, which might store data for you.
Other ways of Inter-Process Communication that can be leveraged to communicate with a database include sockets,
provided through the luasockets
library (requiring an insecure environment and an accessible installation).
If the database server runs on the same machine, you might decide to use file bridges for IPC.
Lightweight database library
Requires an insecure environment and an installation of the database library that is accessible to Minetest.
SQLite3, available through the lsqlite3
luarocks package,
is a popular choice here and used for instance by the sban mod.
String stores
Entity staticdata
Tied to entities. The serialized string must be returned by get_staticdata
and is passed to on_activate
.
File store
Usually tied to world or mod paths. The simplest approach reads the file at load time and writes it on shutdown.
As on_shutdown
may however not be called in the case of a crash -
or even worse, a power outage might abruptly shut down the server without calling anything -
this provides a rather poor granularity, as all changes to the data during the uptime may be lost.
You may simply serialize your data and write it to a file on every update. If your data is rather larger or gets updated frequently, a full serialization might negatively impact performance. Performance can be improved at the expense of granularity by saving periodically and choosing "long" periods. A transaction log improves performance by only storing changes, at the expense of disk space.
TIP: A mix of both approaches can provide satisfying results, logging only changes and rewriting the logfile periodically to keep disk space waste acceptable.
For special cases like logging, an append-only file may be the ideal solution if using the global minetest.log
is not desirable.
Key-value store
Filesystem
On systems that provide a decent filesystem implementation (that is, everything except Windows), you can use filenames/filepaths as keys and files as values. On poor filesystems, you might be heavily limited by absolute path character limits; lots of small files might lead to fragmentation.
A nested hierarchical key-value store is possible through directory structures, which can be managed and traversed using:
minetest.mkdir
minetest.rmdir
minetest.cpdir
minetest.mvdir
minetest.get_dir_list
If you want to mitigate the risk of data loss, you can use minetest.safe_file_write
when (re)writing files.
Configuration files
The Settings
object allows you to operate on configuration files, getting & setting key-value entries and saving the file.
The main Settings
object minetest.settings
can be used to persist a few settings "globally" - bleeding everywhere.
This is horribly abused by the mainmenu to store stuff like the last selected game. Don't be like the mainmenu;
distinguish persistent game data and settings/configuration properly and namespace your data properly.
That said, Settings
can be used to read from & write to simple & limited key-value store files.
The main advantage of them is that they are presumably easy to edit for users,
but this is usually not a requirement for game data which is "edited" by in-game interactions;
indeed, you might not want to tempt players to cheat in singleplayer by editing their "saves".
MetaData
Minetest provides metadata objects which all provide a simple string key-value store, tied to four different game "objects":
- ItemStacks:
ItemStackMetaData
: Fully sent to clients; serialized within inventories, which may be serialized within mapblocks - Node positions:
NodeMetaData
: Sent to clients, but fields can be marked as private; serialized somewhere within mapblocks - Players:
PlayerMetaData
: SQLite-backed key-value storage, however only available while the player is online - Mods:
ModStorage
: SQLite-backed key-value storage; older versions use a JSON-backed store unsuitable for large data (as serialization will block the main thread)
Utilities for setting & getting non-string datatypes like integers and floats are provided; the datatype is however not stored with the entries.
The granularity of all these key-value stores is determined by the server_map_save_interval
setting.
This article is originally based on an article from the minetest_docs project: persistence.adoc by Lars Müller, licensed under CC-BY 4.0