3772: Add client-side HTTP server info r=def- a=heinrich5991
Summary
=======
The idea of this is that clients will not have to ping each server for
server infos which takes long, leaks the client's IP address even to
servers the user does not join and is a DoS vector of the game servers
for attackers.
For the Internet, DDNet and KoG tab, the server list is entirely fetched
from the master server, filtering out servers that don't belong into the
list.
The favorites tab is also supposed to work that way, except for servers
that are marked as "also ping this server if it's not in the master
server list".
The LAN tab continues to broadcast the server info packet to find
servers in the LAN.
How does it work?
=================
The client ships with a list of master server list URLs. On first start,
the client checks which of these work and selects the fastest one.
Querying the server list is a HTTP GET request on that URL. The
response is a JSON document that contains server infos, server addresses
as URLs and an approximate location.
It can also contain a legacy server list which is a list of bare IP
addresses similar to the functionality the old master servers provided
via UDP. This allows us to backtrack on the larger update if it won't
work out.
Lost functionality
==================
(also known as user-visible changes)
Since the client doesn't ping each server in the list anymore, it has no
way of knowing its latency to the servers.
This is alleviated a bit by providing an approximate location for each
server (continent) so the client only has to know its own location for
approximating pings.
## Checklist
- [x] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [ ] Considered possible null pointers and out of bounds array indexing
- [ ] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
Co-authored-by: heinrich5991 <heinrich5991@gmail.com>
Summary
=======
The idea of this is that clients will not have to ping each server for
server infos which takes long, leaks the client's IP address even to
servers the user does not join and is a DoS vector of the game servers
for attackers.
For the Internet, DDNet and KoG tab, the server list is entirely fetched
from the master server, filtering out servers that don't belong into the
list.
The favorites tab is also supposed to work that way, except for servers
that are marked as "also ping this server if it's not in the master
server list".
The LAN tab continues to broadcast the server info packet to find
servers in the LAN.
How does it work?
=================
The client ships with a list of master server list URLs. On first start,
the client checks which of these work and selects the fastest one.
Querying the server list is a HTTP GET request on that URL. The
response is a JSON document that contains server infos, server addresses
as URLs and an approximate location.
It can also contain a legacy server list which is a list of bare IP
addresses similar to the functionality the old master servers provided
via UDP. This allows us to backtrack on the larger update if it won't
work out.
Lost functionality
==================
(also known as user-visible changes)
Since the client doesn't ping each server in the list anymore, it has no
way of knowing its latency to the servers.
This is alleviated a bit by providing an approximate location for each
server (continent) so the client only has to know its own location for
approximating pings.
3779: Copied Teeworld patches for the Haiku operating system r=def- a=panos
I initiated a port of ddnet and received a lot of help from the IRC in order to do so as seen in #2986. Unfortunately, my efforts went stale after one point. Fortunately, however, @threedeyes released a [set of patches for Teeworlds](2ea72967cd/games-action/teeworlds/patches/teeworlds-0.7.5.patchset). I decided to take down my repository with my own work and attempt to port Gerasim's work, which includes code for setting the directories that ddnet should use the same way one would do so in a Mac OS X-system in `src/base/system.c`, which was precisely the part where I got stuck in on my first attempt.
The changes have not been tested thoroughly yet and were the result of methodic copying and pasting.
<!-- What is the motivation for the changes of this pull request -->
## Checklist
- [ ] Tested the change ingame
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [x] Considered possible null pointers and out of bounds array indexing
- [x] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
Co-authored-by: Panagiotis Vasilopoulos <hello@alwayslivid.com>
3568: Get rid of the MySQL C++ connector as a dependency and disable C++ exceptions r=Learath2 a=heinrich5991
We now use the C API directly. This has the advantage of using one
obscure dependency less, as the C++ connector also used the C API.
Also remove exceptions from database code.
The equivalent code with return codes is a human-assisted mechanical
translation of the old code, no attempts to improve correctness were
made.
This allows us to enable the -fno-exceptions flag.
## Checklist
- [x] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [x] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [x] Considered possible null pointers and out of bounds array indexing
- [ ] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
Co-authored-by: heinrich5991 <heinrich5991@gmail.com>
The equivalent code with return codes is a human-assisted mechanical
translation of the old code, no attempts to improve correctness were
made.
This allows us to enable the -fno-exceptions flag.
3614: Mac OS X -> macOS r=heinrich5991 a=def-
Changing the filenames of the releases will be a bit more annoying, so
not sure we want to do that
> Apple shortened the name to "OS X" in 2012 and then changed it to "macOS" in 2016
<!-- What is the motivation for the changes of this pull request -->
## Checklist
- [ ] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [ ] Considered possible null pointers and out of bounds array indexing
- [ ] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
Co-authored-by: def <dennis@felsin9.de>
3492: Support F-Client and replace non-extended extension messages r=Learath2 a=heinrich5991
This continues what #3439 started, and also replaces non-extended messages that are DDRace extensions with extended messages. The DDNet server still sends the old message to old clients and the DDNet client still understands the old messages.
Supersedes #3439.
## Checklist
- [x] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [x] Considered possible null pointers and out of bounds array indexing
- [x] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
3559: Don't highlight us if we have no name (alternative fix for #3554) r=heinrich5991 a=def-
<!-- What is the motivation for the changes of this pull request -->
## Checklist
- [x] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [x] Considered possible null pointers and out of bounds array indexing
- [x] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
3561: Fix input message position in teehistorian files r=heinrich5991 a=Zwelf
Previously the message was recorded when incoming. But the time when the input should apply wasn't recorded. Now inputs are recorded right before they get applied in the game world.
And record exact tick where load/save result is returned.
## Checklist
- [x] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [x] Considered possible null pointers and out of bounds array indexing
- [x] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
3565: Add Arabic language translation by .Yukki r=heinrich5991 a=def-
## Checklist
- [x] Tested the change ingame
- [x] Provided screenshots if it is a visual change
- [x] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [x] Considered possible null pointers and out of bounds array indexing
- [x] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
Co-authored-by: fokkonaut <35420825+fokkonaut@users.noreply.github.com>
Co-authored-by: heinrich5991 <heinrich5991@gmail.com>
Co-authored-by: def <dennis@felsin9.de>
Co-authored-by: Zwelf <zwelf@strct.cc>
3377: Add Thread Safety Analysis r=heinrich5991 a=def-
https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
## Checklist
- [ ] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [ ] Considered possible null pointers and out of bounds array indexing
- [ ] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
3487: Fix centisecs rounding in str_time_float r=heinrich5991 a=def-
<!-- What is the motivation for the changes of this pull request -->
## Checklist
- [ ] Tested the change ingame
- [ ] Provided screenshots if it is a visual change
- [ ] Tested in combination with possibly related configuration options
- [ ] Written a unit test if it works standalone, system.c especially
- [ ] Considered possible null pointers and out of bounds array indexing
- [ ] Changed no physics that affect existing maps
- [ ] Tested the change with [ASan+UBSan or valgrind's memcheck](https://github.com/ddnet/ddnet/#using-addresssanitizer--undefinedbehavioursanitizer-or-valgrinds-memcheck) (optional)
Co-authored-by: def <dennis@felsin9.de>
The canonical way to link with the thread library is to use -pthread, which
brings in additional libraries like libatomic.so on riscv64. However cmake
defaults to link with -lpthread which only bring the libpthread.so library.
Fortunately it has the option THREADS_PREFER_PTHREAD_FLAG for that, which is
"highly recommended" but not the default.
Add `-rdynamic` if supported by the linker.
Not all symbols being exported broke our builds recently, the antibot
library used `mem_comp` from the main binary but it wasn't exported,
leading to the following runtime error:
```
./DDRace64-Server_sql: symbol lookup error: /home/teeworlds/servers/libantibot.so: undefined symbol: mem_comp
```
Declare that we're fine with policies up to 3.19.1. This should fix
policy errors and "too low cmake_minimal_version() errors" for the
forseeable future.
2733: Implement on-demand skin downloader, remove Lappi's non-free skins (fixes#1166, fixes#2547) r=Learath2 a=def-
Seems to work on first try, I'm somewhat surprised :D
Future ideas:
- I haven't checked how the performance impact is of a new skin joining,
might want to move the loading to a separate thread?
- Use the skins from download directory instead of redownloading? Might
want to compare modified time.
- Make all skins load on demand only to save some memory?
Co-authored-by: def <dennis@felsin9.de>
2783: Don't insert timestamps into PEs with MinGW r=def- a=heinrich5991
Should make the build reproducible.
Co-authored-by: heinrich5991 <heinrich5991@gmail.com>
2598: Allow specifying an OpenGL version manually r=def- a=Jupeyy
Ofc don't merge, we'll test it first, and if we want merge it, i'd need to enable shader compability for OpenGL 4+, and general support for OpenGL 3.1 and 3.2(bcs i think they already don't allow some older GL commands).
Also i never really read in the OpenGL 1.x specification, i remember that older OpenGL versions required glBegin and stuff like that
Co-authored-by: Jupeyy <jupjopjap@gmail.com>
2465: Sqlite3 support and prepared statements r=heinrich5991 a=Zwelf
This PR changes the abstraction layer of the score backend to thin abstractions over the MySQL and SQLite3 library. It executes all Queries in one worker thread making it easier to use the ddnet thread pool. This doesn't change much, because each the mysql-connection was locked with `m_SqlLock` beforehand, serializing writes and reads respectively.
Behavior change (even though I tried to minimize them):
* `sv_use_sql` is used to determine if mysql server should be added
* `sv_sql_failure_file` is replaced by `sv_sqlite_file`
* `sv_sqlite_file` is either used as a backup server when `sv_use_sql` is enabled or as the primary read+write server when `sv_use_sql` is disabled
* `/load` now escapes the like-string
Since I am not good at designing config file commands, I would appreciate feedback on this part.
WIP:
* [x] rewrite SQL statements to work in both MySQL and SQLite (preferable just ANSI-SQL)
* [x] create tables (`COLLATE BINARY` and encoding info)
* [x] store rank (UPSERT for points)
* [x] load birthday (different function in sqlite for time handling)
* [x] `/mapinfo` (`convert(? using utf8mb4) COLLATE utf8mb4_general_ci`)
* [x] `/map` (`convert(? using utf8mb4) COLLATE utf8mb4_general_ci`)
* [x] store teamrank (`GROUP_CONCAT`)
* [x] `/teamrank` (`GROUP_CONCAT`)
* [x] ~`/top5team` (`GROUP_CONCAT`)~ doesn't contain GROUP_CONCAT
* [x] `/times` (`UNIX_TIMESTAMP`)
* [x] `/load` without any arguments (`UNIX_TIMESTAMP`)
* [x] all commits compiling, making future bisect easier
* [x] write a sqlite_to_mysql script
* [x] write an old_file_server to sqlite script
* [x] gracefully shutdown DbPool
Co-authored-by: Zwelf <zwelf@strct.cc>
2331: Enable rpath on Linux for shipping SDL2 shared lib r=heinrich5991 a=def-
I guess we should have a way to say when we want to build against the
static sdl2? Or we just ship SDL2 for the ddnet.tw releases as well. I'd
be fine with the latter, but it adds a few MB probably.
Co-authored-by: def <dennis@felsin9.de>
Add libnotify support for linux client
Also unify notification management
Make libnotify mandatory for the client. It is installed on 100% of Arch
Linux systems and on 70% of Debian systems. I'd guess the remaining
Debian systems are servers.
Detect dependent libraries of `libnotify` using `pkg-config`. Remove
library-specific code from the game module. Decrement refcount for
libnotify notification object before leaving the function.
ABI check failed, CMAKE_SIZEOF_VOID_P was thus not set and we look for
libs in wrong directory. Something seems to be wrong with the C/C++
compilers Travis is using. Not sure how to figure out more closely.
1620: Rewrite of prediction code, with additional prediction (updated) r=def- a=trml
This is a reworked attempt at a rewrite of the prediction code (#464), to allow for more client side prediction. It doesn't fix the duplication of server code, but the client code should otherwise be cleaner. This includes separating prediction code out of gamecore/gameclient, and a refactor of the rendering of predicted characters.
There is also prediction for some new things, and some other changes:
- prediction of laser, shotgun, ninja, fng hammer, pickups and ddrace tiles (freeze/unfreeze tiles with cl_predict_freeze)
- laser and bullets are rendered when you fire them and bullets don't go through walls (when both cl_antiping_grenade and cl_antiping_weapons are enabled)
- antiping for flags
- prediction of dummy input
- an additional smoothing option that attempts to make antiping less jumpy (cl_antiping_smooth)
Co-authored-by: trml <trml@users.noreply.github.com>
Co-authored-by: trml <trml@noreply.github.com>
1621: Actually enable -Wuseless-cast r=heinrich5991 a=def-
It's not supported in C and we only checked using a C compiler.
Co-authored-by: Dennis Felsing <dennis@felsin9.de>
- Explanations by Lady Saavik taken from https://ddnet.tw/explain/
- CCW/CW were mixed up in enum names, fixed
- Make sure that the texts fits, otherwise reduce font size
- Still need explanations for Portal tiles
SHA256 was chosen because it is reasonably standard, the file names
don't explode in length (this rules out SHA512) and it is supported by
basically all versions of OpenSSL (this rules out SHA512/256 and SHA3).
The protocol is changed in a backward compatible way: The supporting
server sends the SHA256 corresponding to the map in the `MAP_DETAILS`
message prior to sending the `MAP_CHANGE` message. The client saves the
SHA256 obtained from the `MAP_DETAILS` message until the next
`MAP_CHANGE` message.
For servers not supporting this protocol, the client falls back to
simply opening maps like in the previous scheme.
Remove the `map_version` tool, it is not being used and would have been
a little bit effort to update.
Use the OpenSSL implementation of SHA256 if it is supported, otherwise
fall back to a public domain one.
Fix#1127.
This is done by HTTP POSTing to a location specified by
`sv_modhelp_url`. We also provide a `src/modhelp/server.py` which can
use theses POSTs to forward them to Discord servers.
The POST contains a JSON object payload, with the keys `"port"` which
contains the server port, `"player_id"` which contains the calling
player's client ID, `"player_name"` which contains the calling player's
nick and `"message"` which is the user-specified message.
Make JSON-escaping function public, add tests and fix bugs uncovered by
these tests.
Supersedes #1129.
This uses the Unicode confusable data together with judging how close
two strings are by using the Levenshtein distance.
Adds the commands `name_ban`, `name_unban` and `name_bans`. Kicks
players who join using a banned name and doesn't allow ingame players to
change their names to the banned ones.
Our build system used to put generated files into `src/game/generated`,
now it's using `${PROJECT_BUILD_DIR}/src/game/generated`. Prefer the
ones in `${PROJECT_BUILD_DIR}/src/game/generated`.
947: Add support for extra chunks in teehistorian r=Learath2 a=heinrich5991
This allows to add rarely-used chunks without increasing the file format
version.
978: Disable warnings for GTest r=Learath2 a=heinrich5991
Warnings for GTest broke the build because GTest turns warnings into
errors, which is undesirable if GTest is just used as a dependency.
See also https://github.com/google/googletest/issues/1373.
Warnings for GTest broke the build because GTest turns warnings into
errors, which is undesirable if GTest is just used as a dependency.
See also https://github.com/google/googletest/issues/1373.
941: Add cross-compiled DMG archives r=Learath2 a=heinrich5991
Currently they only contain the client and the archives are a lot larger
than the current release artifacts.
953: Overhauled job system r=Learath2 a=heinrich5991
The engine now takes `std::shared_ptr<IJob>`, this will ensure the
appropriate lifetime of the given parameters, it also allows for proper
destruction. Remove the now obsolete `IFetcher` interface and `CFetcher`
class.
Also adds some locks to `CUpdater`, previously it didn't have any locks
at all.
869: Complete rework of the ghost and race recorder r=Learath2 a=Redix
This PR fixes a lot of issues with the ghost and race recorder, introduces a new ghost file format and implements some pretty useful features.
**A quick overview:**
- When you cross the start line multiple times, both (ghost and recorder) will only restart if it is a non-solo server
- If available, both will use the race timer to recognize the start instead of searching for the start line
- Fastcap support for both
- The recorder immediately starts when the Tee spawns, so the preparation steps (which are quite important for speedruns) will be included in the demo. If you do not cross the start-line within 20 seconds, it will stop the demo until you really start.
- Better ghost menu (colors, reloading, deleting and saving ghosts)
- The ghosts are more resistant against lags (old ones were sometimes completely useless due to small lags)
- New ghosts files are significantly smaller
- Cleanup, bugfixes..
**About the new ghost format (version 4/5):**
This format is used by Teerace for over a year now. The code for handling the binary files was moved to the engine. It includes an auto updater which creates a backup of all ghosts and converts them to the new format afterwards. The major differences from the format used by DDNet right now (version 2), are the ability to store multiple types of data, the usage of delta-encoding and a more portable header.
For whatever reason, the ghost stores values for every predicted tick, but without tick information, so lost snapshots can make them unusable. The new code uses the original values from the snapshots including ticks but it can also handle the old ones without. Since hardly any server uses the high bandwidth option this practically reduces the file size.
Like the demo recorder the ghost recorder directly stores the data to a file (every 50 snapshots) instead of writing the whole file at the end of the race. Indeed this can be changed with only a few lines if the old behavior is preferred.
The updater can handle version 2 (DDNet) and 3 (old teerace format, only slightly different from version 2) files. The updating already happens when the files are scanned for generating the list in the menu and not only when you activate them. The change from version 4 to 5 was only needed due to a bug in the implementation, the ghost loader can read both.
Some numbers about the file size: (map: hotrun, both about 30 seconds)
- Old ghost: 30.4 kB (converted: 10.7 kB)
- New ghost: 5.4 kB
**One thing about the race recorder:**
The old implementation compared the new file only with the first file it found for the particular map. The new one compares with all related demos and deletes them possibly, so that only the best demo is left. Since DDNet can also store the demos without name, this might also delete demos from other players, that you might have in your directory.
To prevent this I at least check whether the demo contains the player name if `cl_demo_name` is on.
In my opinion the better solution would be to remove `cl_demo_name` and always use the player name.
teehistorian records all inputs from the players as well as the player
positions in each tick. It stores this info in a highly compressible
output format (I've achived 5x compression using xz or bz2).
On Linux, `install` installs DDNet into the normal file system (root
rights required).
`package` tries to create an archive similar to today's release archives
and `package_source` tries to create a source archive similar to today's
source archives.
- Single json file containing all information for the client
- Fetched from https://info.ddnet.tw/info?name=deen
- Replaces versionsrv, news, ddnet-maps.json and ddnet-ranks.json
- Servers are sorted by most popular ones for respective player
- Always stays < 100 ms response time, compared to occasional 50 s for
old ddnet-ranks.json
Edgar
ChillerDragon
