Using `CSnapshotStorage::CHolder`s in the `demo_extract_chat` tool is not necessary, as only the member variable `m_pAltSnap` is being used in this tool. The regular snapshot data was copied into `m_pSnap` but never read. The other member variables were initialized but never read.
Reduce the memory footprint of the tool by unloading the map data as soon as possible instead of only implicitly when the `CDataFileReader` is destructed.
Use `log_info` and `log_error` instead of `dbg_msg`.
Add error messages when output files cannot be opened for writing.
Move log messages regarding files being written so they are only printed when the files are actually being written.
Read the entire file into memory immediately when the line reader is initialized instead of using a fixed size buffer of size 32769, which leads to broken line reading for larger files (closes#8431).
Replace the `CLineReader::Init` function with the `CLineReader::OpenFile` function, which additionally checks whether the file contains any null bytes (indicates that the file is likely not a text file).
As the file will be read into memory entirely in the `OpenFile` function, is can also be closed immediately, since using the `io_read_all_str` function should ensure that nothing more can be read from the file. This also simplifies the usage of the `CLineReader` class, as manually closing the file is inconvenient and error-prone. In fact, the file handle for the `ddnet-serverlist-urls.cfg` file was previously not closed properly.
Benchmarking on Windows did not show any noticeable performance impact of this change both for smaller files (config and language files) and larger files (synthetic ~10 MiB files).
Let the `CLineReader::Get` function return `const char *` instead of `char *`, since users of this class should not modify the internal buffer. Also, since the entire file is read into memory now, the returned strings are now valid until the respective line reader object is destructed, instead of only until the `Get` function is called again. This simplifies the usage in some cases where all lines are handled, since additional temporary buffers are now unnecessary.
Remove the `IOFLAG_SKIP_BOM` flag from the `io_open` function, as this flag was only used together with the line reader. This was inconvenient to use, as any use of the `io_open` function specifically for line readers had to be used with `IOFLAG_SKIP_BOM`. Now, the line reader transparently skips the UTF-8 BOM internally. Skipping the UTF-8 BOM never worked with the `IStorage::ReadFileStr` function, because `io_length` is used in the `io_read_all` function, which rewinds the file position to before the UTF-8 BOM that was skipped by using `IOFLAG_SKIP_BOM`. In any case, as the `ReadFileStr` function is currently unused, this has/had no further effect. The respective test cases for `IOFLAG_SKIP_BOM` are removed.
Add more test cases for the `CLineReader` class. All test cases are checked both with and without the UTF-8 BOM. Additional tests with mixed new lines, empty lines with different new lines, and internal null bytes are added.
Consistently use the construct `while(const char *pLine = LineReader.Get())` to iterate over all lines of a line reader. In this case, the assignment inside the `while`-loop seems acceptable over the alternatives, e.g. `while(true)` with `break` or adding a function `CLineReader::ForAll(std::function<const char *> Consumer)`.
Add strict validation for `StrToInts` function. Because this function should only be used with trusted internal strings, assertions are added to ensure that the string is not truncated. Some buffer sizes are adjusted accordingly, so truncation cannot happen.
Add less strict validation for `IntsToStr` function. An additional argument specifying the size of the output buffer is added to assert that the size of the output buffer is sufficient. However, because this function is used to decode data sent by the server and read from maps and ghosts, invalid input data should never result in crashes or invalid UTF-8 strings. The function will now unpack an empty string and return `false`, if the string contains invalid UTF-8.
The inline definition of the functions is not wanted, because it requires adding a `system.h` include in `gamecore.h`. Therefore the tools now have to depend on game-shared, which previously only worked because the functions were inline.
Tests are added to ensure the function still behaves the same as before for valid inputs and correctly handles invalid inputs.
Previously, usage of `void *`, `unsigned char *` and `uint8_t *` was mixed in various places for pointers to raw image data and the pointers ended up being cast to `uint8_t *` at some point anyway. Now only `uint8_t *` is used consistently, which improves type safety and readability. Casts to `uint8_t *` are now only necessary when using `malloc` or when reading data from a map.
When empty PNG files are loaded, the `std::vector` for the file contents is resized to size 0, which results in undefined behavior when it is accessed with `front`.
When `io_tell` fails, i.e. returns `-1`, this was incorrectly cast to an `unsigned` and therefore caused a very large allocation and potentially crashes due to lack of memory.
Use [virtual terminal processing](https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences) for Windows console output when possible, using the existing async logger. Fallback to the old Windows logger when virtual terminal processing is not available. Do not set any logger when there is no console/file/pipe to output to.
This adds support for 24 bit RGB colors using ANSI escape codes (when Terminal emulator supports it) instead of only supporting 16 fixed colors.
Use the existing async logger also on Windows by converting the console standard output handle to an IOHANDLE. Avoid converting console output to UTF-16 by changing the console output codepage to UTF-8 and outputting that directly. Overall, this reduces the time until the loading screen first appears when launching the client from the console on Windows by around 70%, from 80-100ms down to 25ms.
Add support for `NO_COLOR` environment variable also on Windows.
Use the async logger also for writing output to files and pipes instead of using a Windows specific implementation for this, to reduce lags when redirecting the standard output.
Closes#3925. The client/server will not freeze anymore when entering selection mode in the console because we are using the async logger on Windows now.
Closes#1108. We were already using AIO on Windows to write the logfile, which was not causing any known issues. We are now also using AIO with the async loggers for console/files/pipes on Windows.
By adding `CDataFileWriter::ECompressionLevel` to replace usage of zlib internal compression levels in the `CDataFileWriter` API.
Use `std::numeric_limits<int>::max()` instead of `INT_MAX` in one case where the latter was only declared by the transitive zlib include. The `limits` header is already included and its use is more fitting for C++ code.
Adapt the `CDataFileReader::GetItem` function so it optionally also returns a `CUuid` for UUID-based map items, including for items with unknown UUIDs where the item type will be `-1`. Adapt the `CDataFileWriter::AddItem` function so it optionally also accepts a `CUuid` which will be used if the item type is `-1`.
The additional checks for invalid map item types in the map tools are removed again and instead the new UUID parameters are used so map items with unknown UUIDs are written back to maps correctly.
Closes#7701.
All `map_*` tools were crashing with the assertion `Invalid type` when used on maps that contain unknown UUID-based map items. When the UUID cannot be resolved, the type `-1` is returned by the `GetItem` function. The assertion predates UUID map items, so this crash has likely existed since the introduction of UUID map items. The editor was not affected and has instead always discarded map items that it does not support. The tools will now also discard map items with unknown UUIDs.
See #7669.
Do not use the `CMapItemSound::m_SoundDataSize` value as it is redundant. This value could also be incorrect because it can be freely set by the map creator (tool).
Instead, use the map/datafile function `GetDataSize` to get the true size of the sound data in the file.
The `m_SoundDataSize` value is still written to map files for compatibility with old versions.
Replace unnecessary `gameclient.h` include with more specific includes.
Fix storage creation error message not being logged as the logger was initialized after checking for the failed storage creation. However, in this case we want to avoid non-error log messages so the tool's output is only the extracted demo chat, except in error cases.
Rename `Process` function to `ExtractDemoChat` and make it `static` to avoid exporting it.
Use `log_error` instead of `dbg_msg`.
Limit number of sounds being loaded in the client to 64, instead of writing sample indices out of bounds.
Check if sound indices are out of bounds. It was only checked whether the sound index is -1 but no other bounds checking was performed.
Check if image indices are out of bounds. It was not checked for all negative image indices, only -1 was considered.
Add popup message in the editor when trying to add more than 64 sounds, same as for images.
Limit number of images in `map_convert_07` tool, instead of writing out of bounds.
The map version was only checked if the version item is present, which is different from how client and editor load maps. Now a missing version item is considered an unsupported version. Additional log messages are also added to the tools.
Simplify the usage of datafile reader and writer by adding utility functions to read and write zero-terminated UTF-8 strings.
Improve validation of string data read from datafiles. It is ensure that string data is null-terminated, has no internal NUL-characters and is valid UTF-8.
Fix loading of external sounds in the editor. The wrong path variable was being used, so the sound files would not be loaded from correct folder.
Add tests for new datafile reader/writer functions.
The pixel size (bytes per pixel) always has to be 4 for the `Dilate` function to work correctly. This is already checked before calling the function, so the redundant argument which is always `4` can be removed.
Use `enum EImageFormat` type for image format literals and variables.
Add `PixelSize` function to get the number of bytes/color channels per pixel for a specified image format.
Remove unused store format argument of texture loading functions. All textures are automatically being stored as RGBA, so the argument was unused. Also remove the therefore unused `FORMAT_AUTO`.
Rename variables consistently to `PixelSize` and use `size_t`, instead of mixing different names like `BPP` and `ColorChannelCount`.
Validate image format loaded from maps using `CImageInfo::ImageFormatFromInt`. Add `FORMAT_ERROR` to represent invalid formats.
Remove redundant `PixelSize` parameter from graphics backends and commands, which can be derived from the texture format.
Fix memory leak when RGB image data is being converted to RGBA format when saving map in editor.
ChillerDragon
