Read the entire file into memory immediately when the line reader is initialized instead of using a fixed size buffer of size 32769, which leads to broken line reading for larger files (closes#8431).
Replace the `CLineReader::Init` function with the `CLineReader::OpenFile` function, which additionally checks whether the file contains any null bytes (indicates that the file is likely not a text file).
As the file will be read into memory entirely in the `OpenFile` function, is can also be closed immediately, since using the `io_read_all_str` function should ensure that nothing more can be read from the file. This also simplifies the usage of the `CLineReader` class, as manually closing the file is inconvenient and error-prone. In fact, the file handle for the `ddnet-serverlist-urls.cfg` file was previously not closed properly.
Benchmarking on Windows did not show any noticeable performance impact of this change both for smaller files (config and language files) and larger files (synthetic ~10 MiB files).
Let the `CLineReader::Get` function return `const char *` instead of `char *`, since users of this class should not modify the internal buffer. Also, since the entire file is read into memory now, the returned strings are now valid until the respective line reader object is destructed, instead of only until the `Get` function is called again. This simplifies the usage in some cases where all lines are handled, since additional temporary buffers are now unnecessary.
Remove the `IOFLAG_SKIP_BOM` flag from the `io_open` function, as this flag was only used together with the line reader. This was inconvenient to use, as any use of the `io_open` function specifically for line readers had to be used with `IOFLAG_SKIP_BOM`. Now, the line reader transparently skips the UTF-8 BOM internally. Skipping the UTF-8 BOM never worked with the `IStorage::ReadFileStr` function, because `io_length` is used in the `io_read_all` function, which rewinds the file position to before the UTF-8 BOM that was skipped by using `IOFLAG_SKIP_BOM`. In any case, as the `ReadFileStr` function is currently unused, this has/had no further effect. The respective test cases for `IOFLAG_SKIP_BOM` are removed.
Add more test cases for the `CLineReader` class. All test cases are checked both with and without the UTF-8 BOM. Additional tests with mixed new lines, empty lines with different new lines, and internal null bytes are added.
Consistently use the construct `while(const char *pLine = LineReader.Get())` to iterate over all lines of a line reader. In this case, the assignment inside the `while`-loop seems acceptable over the alternatives, e.g. `while(true)` with `break` or adding a function `CLineReader::ForAll(std::function<const char *> Consumer)`.
The `unknown address` log message got printed very often because many servers have 0.6 and 0.7 addresses but only 0.6 addresses are considered valid by the client.
Add `IJob::Abortable(bool)` function which jobs can call to specify whether they can be aborted. Jobs are not abortable per default. Abortable jobs may have their state set to `IJob::STATE_ABORTED` at any point if the job was aborted. The job state should be checked periodically in the `IJob::Run` function and the job should terminate at the earliest, safe opportunity when aborted. Scheduled jobs which are not abortable are guaranteed to fully complete before the job pool is shut down. However, if the job pool is already shutting down, no additional jobs will be enqueue anymore and abortable jobs will immediately be aborted.
In particular, the sound loading, community icon loading, master chooser and host lookup jobs are specified as being abortable. Conversely, the jobs saving replay demos, editor maps and screenshots are expected to finish before the client is shut down.
When the client is quitting/restarting, it will now disconnect from the current server first, before saving the config, to ensure that any actions that happen on disconnect (demo recorders being stopped etc.) happen first. The shutdown message is rendered before disconnecting and waiting for background jobs to finish.
The HTTP client is now initialized later during server launch, after the network initialization. Error handling is added and the server stops if the HTTP client could not be initialized, same as the client.
The `RunBlocking` functions are removed, as they are not used anymore after curl-multi was added.
The function `IJob::Status` is renamed to `State` and `IJob::STATE_PENDING` is renamed to `STATE_QUEUED` for consistency with naming of the HTTP client.
The member variables of the engine interface are encapsulated and the `jobs.h` include is removed from `engine.h`, which removes transitive includes of `system.h`.
Documentation for all job and job pool API is added.
Previously, it was assumed `ResultJson` would return `nullptr` for HTTP requests which have been aborted or failed with an error, which would trigger the newly added assertion error "Request not done".
Replace usages of platform specific `lock_*` functions with `std::mutex` through the wrapper class `CLock`. Move lock classes to `base/lock.h`.
The `CLock` wrapper class is only necessary because the clang thread-safety attributes are not available for `std::mutex` except when explicitly using libc++.
This still works
$ ./DDNet "connect 127.0.0.1"
But now also this works
$ ./DDNet "connect tw-0.6+udp://ger10.ddnet.org:8303"
2023-07-16 14:07:50 I engine: running on unix-linux-amd64
2023-07-16 14:07:50 I client: starting...
2023-07-16 14:07:50 I client: version 17.1.1 on linux amd64
2023-07-16 14:07:50 I client: git revision hash: 7f100e2620
2023-07-16 14:07:50 I client: connecting to 'tw-0.6+udp://ger10.ddnet.org:8303'
2023-07-16 14:07:50 I host_lookup: host='ger10.ddnet.org' port=8303 3
2023-07-16 14:07:51 I client: connected, sending info
Support is incomplete for `leak_ip_address_to_all_servers` (will only
ping the first address of each server) and for the `leak_ip` setting
(which will also only ping the first address of each server).
It didn't have a clear role, it just acted as a distinguisher between
two functions with the same name.
Rename `tw::time_get` to `time_get_nanoseconds` and delete the old
`time_get_nanoseconds`. Move `CCmdlineFix` and the typed
`net_socket_read_wait` function to the global namespace.
5260: Pr thread safety negative r=heinrich5991 a=def-
WorkerThread is hard because `REQUIRES(!((CJobPool *)pUser)->m_Lock)` would require alias analysis or the function using that everywhere.
https://clang.llvm.org/docs/ThreadSafetyAnalysis.html#negative
## Checklist
- [ ] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [ ] Considered possible null pointers and out of bounds array indexing
- [ ] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
Co-authored-by: Dennis Felsing <dennis@felsin9.de>
Change filename `serverlist_urls.cfg` → `ddnet-serverlist-urls.cfg`,
change filename `cache.sqlite3` → `ddnet-cache.sqlite3`, add `const` to
a variable and change pings to just before multiples of 100.
Fixes#3853.
Summary
=======
The idea of this is that clients will not have to ping each server for
server infos which takes long, leaks the client's IP address even to
servers the user does not join and is a DoS vector of the game servers
for attackers.
For the Internet, DDNet and KoG tab, the server list is entirely fetched
from the master server, filtering out servers that don't belong into the
list.
The favorites tab is also supposed to work that way, except for servers
that are marked as "also ping this server if it's not in the master
server list".
The LAN tab continues to broadcast the server info packet to find
servers in the LAN.
How does it work?
=================
The client ships with a list of master server list URLs. On first start,
the client checks which of these work and selects the fastest one.
Querying the server list is a HTTP GET request on that URL. The
response is a JSON document that contains server infos, server addresses
as URLs and an approximate location.
It can also contain a legacy server list which is a list of bare IP
addresses similar to the functionality the old master servers provided
via UDP. This allows us to backtrack on the larger update if it won't
work out.
Lost functionality
==================
(also known as user-visible changes)
Since the client doesn't ping each server in the list anymore, it has no
way of knowing its latency to the servers.
This is alleviated a bit by providing an approximate location for each
server (continent) so the client only has to know its own location for
approximating pings.