Read the entire file into memory immediately when the line reader is initialized instead of using a fixed size buffer of size 32769, which leads to broken line reading for larger files (closes#8431).
Replace the `CLineReader::Init` function with the `CLineReader::OpenFile` function, which additionally checks whether the file contains any null bytes (indicates that the file is likely not a text file).
As the file will be read into memory entirely in the `OpenFile` function, is can also be closed immediately, since using the `io_read_all_str` function should ensure that nothing more can be read from the file. This also simplifies the usage of the `CLineReader` class, as manually closing the file is inconvenient and error-prone. In fact, the file handle for the `ddnet-serverlist-urls.cfg` file was previously not closed properly.
Benchmarking on Windows did not show any noticeable performance impact of this change both for smaller files (config and language files) and larger files (synthetic ~10 MiB files).
Let the `CLineReader::Get` function return `const char *` instead of `char *`, since users of this class should not modify the internal buffer. Also, since the entire file is read into memory now, the returned strings are now valid until the respective line reader object is destructed, instead of only until the `Get` function is called again. This simplifies the usage in some cases where all lines are handled, since additional temporary buffers are now unnecessary.
Remove the `IOFLAG_SKIP_BOM` flag from the `io_open` function, as this flag was only used together with the line reader. This was inconvenient to use, as any use of the `io_open` function specifically for line readers had to be used with `IOFLAG_SKIP_BOM`. Now, the line reader transparently skips the UTF-8 BOM internally. Skipping the UTF-8 BOM never worked with the `IStorage::ReadFileStr` function, because `io_length` is used in the `io_read_all` function, which rewinds the file position to before the UTF-8 BOM that was skipped by using `IOFLAG_SKIP_BOM`. In any case, as the `ReadFileStr` function is currently unused, this has/had no further effect. The respective test cases for `IOFLAG_SKIP_BOM` are removed.
Add more test cases for the `CLineReader` class. All test cases are checked both with and without the UTF-8 BOM. Additional tests with mixed new lines, empty lines with different new lines, and internal null bytes are added.
Consistently use the construct `while(const char *pLine = LineReader.Get())` to iterate over all lines of a line reader. In this case, the assignment inside the `while`-loop seems acceptable over the alternatives, e.g. `while(true)` with `break` or adding a function `CLineReader::ForAll(std::function<const char *> Consumer)`.
Add strict validation for `StrToInts` function. Because this function should only be used with trusted internal strings, assertions are added to ensure that the string is not truncated. Some buffer sizes are adjusted accordingly, so truncation cannot happen.
Add less strict validation for `IntsToStr` function. An additional argument specifying the size of the output buffer is added to assert that the size of the output buffer is sufficient. However, because this function is used to decode data sent by the server and read from maps and ghosts, invalid input data should never result in crashes or invalid UTF-8 strings. The function will now unpack an empty string and return `false`, if the string contains invalid UTF-8.
The inline definition of the functions is not wanted, because it requires adding a `system.h` include in `gamecore.h`. Therefore the tools now have to depend on game-shared, which previously only worked because the functions were inline.
Tests are added to ensure the function still behaves the same as before for valid inputs and correctly handles invalid inputs.
Add `IJob::Abortable(bool)` function which jobs can call to specify whether they can be aborted. Jobs are not abortable per default. Abortable jobs may have their state set to `IJob::STATE_ABORTED` at any point if the job was aborted. The job state should be checked periodically in the `IJob::Run` function and the job should terminate at the earliest, safe opportunity when aborted. Scheduled jobs which are not abortable are guaranteed to fully complete before the job pool is shut down. However, if the job pool is already shutting down, no additional jobs will be enqueue anymore and abortable jobs will immediately be aborted.
In particular, the sound loading, community icon loading, master chooser and host lookup jobs are specified as being abortable. Conversely, the jobs saving replay demos, editor maps and screenshots are expected to finish before the client is shut down.
When the client is quitting/restarting, it will now disconnect from the current server first, before saving the config, to ensure that any actions that happen on disconnect (demo recorders being stopped etc.) happen first. The shutdown message is rendered before disconnecting and waiting for background jobs to finish.
The HTTP client is now initialized later during server launch, after the network initialization. Error handling is added and the server stops if the HTTP client could not be initialized, same as the client.
The `RunBlocking` functions are removed, as they are not used anymore after curl-multi was added.
The function `IJob::Status` is renamed to `State` and `IJob::STATE_PENDING` is renamed to `STATE_QUEUED` for consistency with naming of the HTTP client.
The member variables of the engine interface are encapsulated and the `jobs.h` include is removed from `engine.h`, which removes transitive includes of `system.h`.
Documentation for all job and job pool API is added.
Files were not being truncated on Windows anymore when using `io_open` with `IOFLAG_WRITE` due to a regression from #7254. Instead, the existing file contents were kept and the file pointer was set to the beginning of the file.
This caused broken demo files to be created (#7349) when recording a shorter demo with the same filename as an existing longer demo. It also caused the map tools to produce maps with additional junk data at the end, if an existing map is overridden by a smaller map.
This is fixed by using the creation disposition `CREATE_ALWAYS` instead of `OPEN_EXISTING` with [`CreateFileW`](https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew), which ensures that a file is always created and truncated.
A regression test case is added, which fails without this change.
Add stricter error handling when converting between UTF-16 (wide characters) and UTF-8 (multi-byte) on Windows.
The `windows_wide_to_utf8` function now returns an `std::optional`, which will be empty if the argument contains invalid UTF-16. Files/folders with names containing invalid UTF-16 are now ignored on Windows. It was previously not possible to use these files either, as converting their names to UTF-8 changed the invalid codepoints to unicode replacement characters.
The `windows_utf8_to_wide` function now fails with an assertion error if the argument contains invalid UTF-8, as this should never happen.
Closes#7486.
Unify all code for name bans in new class `CNameBans` in the existing `name_ban.cpp/h` files. The previously global function `IsNameBanned` is now the member function `CNameBans::IsBanned`. The existing name ban tests are extended for the `CNameBans` class.
Move `CNameBan` constructor definition to source file to avoid including `system.h` in the header file. Use `bool` instead of `int` for `m_IsSubstring`. Reorder `CNameBan` constructor arguments and remove unnecessary default value.
Initialize nontrivial types with a constructor instead. Make the
compiler aware that some of our constructors are indeed trivial.
This allows `mem_zero` calls to actually always zero the memory.
Partially replaces #5690.
POD types are just memset. Other types are either destructed if not
trivial and/or constructed if not trivial. Types need to have a default
constructor.
Virtual classes can be mem_zeroed only if they already have been
constructed, otherwise it is UB.
Replace usages of platform specific `lock_*` functions with `std::mutex` through the wrapper class `CLock`. Move lock classes to `base/lock.h`.
The `CLock` wrapper class is only necessary because the clang thread-safety attributes are not available for `std::mutex` except when explicitly using libc++.
Round to nearest integer instead of truncating in `f2fx` to ensure correct round-trip with `fx2f`.
Add test to ensure correct round-trip with maximum `0.0005f` absolute error.
The host lookup job and the engine interface are independent so they are moved to separate files.
The include of `engine.h` in `client.h` is therefore unnecessary and other includes also had to be adjusted because of this.
The variable `m_VersionServeraddr` is unused and therefore removed. The host lookup job is currently not used on the client-side.
Simplify the usage of datafile reader and writer by adding utility functions to read and write zero-terminated UTF-8 strings.
Improve validation of string data read from datafiles. It is ensure that string data is null-terminated, has no internal NUL-characters and is valid UTF-8.
Fix loading of external sounds in the editor. The wrong path variable was being used, so the sound files would not be loaded from correct folder.
Add tests for new datafile reader/writer functions.
vanilla 0.6 joins weren't recorded. Also after map change, for the
existing players these Join chunks were missing. Just add these
message for existing players now and add OnClientEngineJoin in all
code paths with clients connecting.
Add `str_utf8_offset_bytes_to_chars` and `str_utf8_offset_chars_to_bytes` functions to base system to convert between byte and UTF-8 character offsets in UTF-8 strings.
Previously, this was separately implemented in the textrender and in the lineinput helper.
These textrender functions are entirely replaced by the new functions:
- `ITextRender::SelectionToUTF8OffSets` (by `str_utf8_offset_chars_to_bytes`)
- `ITextRender::UTF8OffToDecodedOff` (by `str_utf8_offset_bytes_to_chars`)
- `ITextRender::DecodedOffToUTF8Off` (by `str_utf8_offset_chars_to_bytes`)
These lineinput helper functions are reimplemented using the new functions:
- `CLineInput::OffsetFromActualToDisplay` (uses `str_utf8_offset_bytes_to_chars`)
- `CLineInput::OffsetFromDisplayToActual` (uses `str_utf8_offset_chars_to_bytes`)
This still works
$ ./DDNet "connect 127.0.0.1"
But now also this works
$ ./DDNet "connect tw-0.6+udp://ger10.ddnet.org:8303"
2023-07-16 14:07:50 I engine: running on unix-linux-amd64
2023-07-16 14:07:50 I client: starting...
2023-07-16 14:07:50 I client: version 17.1.1 on linux amd64
2023-07-16 14:07:50 I client: git revision hash: 7f100e2620
2023-07-16 14:07:50 I client: connecting to 'tw-0.6+udp://ger10.ddnet.org:8303'
2023-07-16 14:07:50 I host_lookup: host='ger10.ddnet.org' port=8303 3
2023-07-16 14:07:51 I client: connected, sending info
Port the `CJsonWriter` utility class from upstream, which makes outputting correct JSON easier.
Add `CJsonWriter` as an abstract class that can write to different outputs. Two implementations `CJsonFileWriter` (writes to a file) and `CJsonStringWriter` (writes to an `std::string`) are added. Upstream `CJsonWriter` can only write to files.
The same tests are added for both implementations. Duplicate code is avoided by using typed tests with two separate test fixtures.
Typed tests have test names like "TestName/0" and "TestName/1", which would result in invalid filenames. Replace the string after the first slash with the name of the typed test and use hyphen instead of slash.
Add optional `end` output parameter to `str_utf8_find_nocase` which is set to a pointer into the haystack after the matched string.
This is necessary because there are pairs of matching upper case and lower case Unicode characters with different byte length (e.g. both `I` and `İ` map to `i`), so the byte length of the string matched in the haystack may not be identical to the length of the needle string.
ChillerDragon
