Another prolific year of open-source contributions
This time last year, we wrote about the more than 190 Trail of Bits-authored pull requests that were merged into non-Trail of Bits repositories in 2021. In 2022, we continued that trend by having more than 400 pull requests merged into non-Trail of Bits repositories!
Why is this significant? While we take great pride in the tools that we develop, we recognize that we benefit from tools maintained outside of Trail of Bits. When one of those tools doesn’t work as we expect, we try to fix it. When a tool doesn’t fill the need we think it was meant to, we try to improve it. In short, we try to give back to the community that gives so much to us.
Here are a few highlights from the list of PRs at the end of this blog post:
- Clippy is a collection of over 550 lints to catch common mistakes and improve Rust code. We added the crate_in_macro_def and unnecessary_find_map lints, and contributed improvements and bugfixes to lints such as empty_line_after_outer_attribute, expect_used/unwrap_used, extra_unused_lifetimes, needless_borrow, needless_lifetimes, unnecessary_to_owned, and unnecessary_filter_map.
- HEVM is an implementation of the Ethereum virtual machine with symbolic execution capabilities. Our contributions to HEVM included simplifying its use of the SMT solver, improving its performance, fixing a memory leak, and adding tests.
- Envoy is a high-performance open source edge and service proxy that makes the network transparent to applications. We implemented the initial version of the Unified Header Validation (UHV) component within Envoy for validating all request and response headers for HTTP/1 and HTTP/2. We took the existing header validation logic, consolidated it into the UHV component, performed an assessment to determine where the logic was not fully RFC compliant, and then fixed or implemented any gaps to ensure that the default configuration strictly adheres to the RFC standards. The new component provides a single entry point for all HTTP request and response validation that makes it a much easier code base to maintain, audit, extend, customize, and fix any newly discovered attack vectors.
- pyca/cryptography is a package that provides cryptographic recipes and primitives to Python developers. We improved its support for Certificate Transparency and made numerous usability improvements.
- Vcpkg is a C/C++ package manager for Windows, Linux, and MacOS. We fixed a bug in Vcpkg itself and made improvements to packages such as flatbuffers, grpc, gtest, ixwebsocket. libcpplocate, llvm, mbedtls, tcb-span, and z3.
- Warehouse is the software that powers PyPI, the official package index for the Python programming language. We made numerous feature improvements and bugfixes, including support for expiring API tokens, support for credential-free package uploads with OIDC, a refactor of core permissions internals, enhancements to PyPI’s vulnerability feed, and improvements to user-facing error messages.
The projects named below represent software of the highest quality. Software of this caliber doesn’t come from just merging PRs and publishing new releases; it comes from careful planning, prioritizing features, familiarity with related projects, and an understanding of the role that a project plays within the larger software ecosystem. We thank these projects’ maintainers both for the work the public sees and for innumerable hours spent on work the public doesn’t see.
We wish you a happy, safe, and similarly productive 2023!
Some of Trail of Bits’s 2022 Open-Source Contributions
Cryptography
Tech Infrastructure
- abodelot/jquery.json-viewer
- aio-libs/aiohttp
- curl/curl
- curl/curl-fuzzer
- di/bump
- di/pip-api
- GaloisInc/daedalus
- googleapis/google-auth-library-python
- Homebrew/homebrew-core
- solc-select 0.2.1 (new formula) #107977
- crytic-compile 0.2.3 (new formula) #108010
- slither-analyzer 0.8.3 (new formula) #108016
- echidna 2.0.2 (new formula) #108045
- echidna 2.0.3 #110107
- crytic-compile 0.2.4 #112418
- slither-analyzer 0.9.0 #112423
- solc-select 1.0.1 #112793
- slither-analyzer 0.9.1 #114631
- echidna 2.0.4 #116508
- iovisor/ubpf
- killercup/cargo-edit
- llvm/llvm-project
- microsoft/ebpf-for-windows
- microsoft/vcpkg
- [libcpplocate] New port #23173
- [tcb-span] Add new port #23393
- [llvm] Fix LLVM install for ‘utils’ feature #23399
- [ixwebsocket] Update to v11.3.3 #23548
- [gtest] Remove -Werror #23780
- [mbedtls] Update to latest 2.x LTS version #23787
- [flatbuffers] Update to 2.0.6 #24208
- [z3] Update to 4.8.15 #24209
- [z3] Update to 4.8.16 #24407
- [grpc] Fix path quoting #24948
- [mbedtls] Update to v2.28.1 #25894
- [z3] Update to v4.9.1 #25911
- [z3] Update to v4.10.2 #25954
- [grpc] Fix protobuf protoc executable variable #26199
- [vcpkg] Fix cross compiling macOS #26240
- [z3] Update to 4.11.0 #26429
- NixOS/nixpkgs
- haskellPackages.clash-prelude: fix build by disabling tests #178868
- semgrep: 0.106.0 -> 0.108.0 #185771
- echidna: 1.7.3 -> 2.0.2 #190144
- echidna: 2.0.2 -> 2.0.3 #190775
- libff: dynamic by default #190786
- xed: 12.0.1 -> 2022.08.11 #191045
- echidna: 2.0.3 -> 2.0.4 #202542
- haskellPackages: remove unnecessary overrides from ghc-9.2.x #202604
- haskellPackages: configuration cleanup #202615
- haskellPackages: configuration common cleanup #203180
- haskellPackages: unbreak selected packages #203327
- haskellPackages: unbreak selected packages #203489
- nodejs/node
- Add script for vulnerability checking of Node.js dependencies #43362
- Add support for using API key to vuln checking script #43909
- Update
undiciCPE in vulnerability checking script #44128 - Add automation for updating
base64dependency #45300 - Add automation for updating
acorndependency #45357 - Add automation for updating
libuvdependency #45362 - tools: add missing step in update-base64.sh script #45509
- Remove dependency vulnerability checker #45675
- osquery/osquery
- Fix check for PIE support #7234
- Improve Pidfile handling #7304
- Prevent the audit event system from using too much memory #7329
- Fix globToRegex truncating UTF16 characters #7430
- Do not run clang-tidy on third party libraries #7432
- Change the JSON of the results from an event scheduled query to an array #7434
- Add new metrics and improve description of existing ones in osquery_schedule #7438
- Fix a crash when Yara uses its own strutils functions #7439
- bpf: Improve socket event handling #7446
- Update cppcheck to version 2.6.3 and skip analysis for third party code #7455
- Fix submodule cache for macOS CI runner #7456
- Add third party libraries target #7467
- Add BOOST_USE_ASAN define when enabling Asan #7469
- Enable fuzzing and Asan on Windows, enable Asan on macOS #7470
- Fix user_time and system_time unit in processes table on M1 #7473
- Fix watchdog not killing unhealthy worker/extension fast enough #7474
- Fix some warnings about unrecognized special characters #7478
- yum_sources: Include the mirrorlist URL in the table output #7479
- Fix third party libraries flags leaking to osquery targets #7480
- Replace WmiRequest constructor with static factory method #7489
- Change cpu_info test to ensure *at least* one socket is present #7490
- Improve scheduled query denylisting and scheduler shutdown #7492
- Fix the test_http_server.py –persist option #7497
- bpf: Disable the BPF publisher in case of error #7500
- Mark wall_time column in osquery_schedule as hidden #7501
- Add a mechanism to reduce memory retained on Linux #7502
- Fix crash due to interaction between distributed and config plugin #7504
- libs: Update OpenSSL from version 1.1.1l to 1.1.1n #7506
- Implement a performant cache for users and groups on Windows #7516
- Remove libelfin and elf parsing tables #7524
- Update librpm to 4.17.0 #7529
- Eliminate removal of nonblocking flag for “special” files #7530
- Fixes to unblock the CI #7533
- Drop shortcut_files table #7547
- libs: Update zlib from v1.2.11 to v1.2.12 #7548
- libs: Update libdpkg from version v1.19.0.5 to v1.21.7 #7549
- Prevent ebpfpub linking against the system zlib #7557
- Restore some release checks #7558
- Add an option to specify a path to the openssl archive #7559
- Prevent CLI_FLAGs to be set via config #7561
- Change where the macOS Info.plist is generated #7566
- Fix DebPackages.test_sanity test #7569
- certificates table: Add Linux support #7570
- CHANGELOG 5.2.3 #7571
- Fix release tests for Linux aarch64 #7572
- Use additional instead of index for admindir in deb_packages #7573
- CHANGELOG 5.3.0 #7575
- Explicitly set context for the tables reading utmpx databases #7578
- certificates: Refactor the OpenSSL utilities #7581
- Warn about setting CLI_FLAGs in the config #7583
- Remove the test_daemon_sighup test #7584
- Restore macOS
kernel_panicstable on modern macOS #7585 - Replace
OS XwithmacOSin table specs #7587 - Fix MBCS support on Windows #7593
- Remove CLI flags settings from osquery.example.conf #7595
- Correct the section where the users and groups service flags are described #7596
- Fix shared_resources accessing uninitialized variables #7600
- Remove redundant string conversion #7603
- Update the “new release” issue template #7607
- Fix a UUID typo in the
disk_encryptiontable #7608 - Add an option to build with the leak sanitizer #7609
- Fix SchedulerTests.test_scheduler_drift_accumulation flakyness #7613
- Fix multiple Yara leaks #7615
- Initialize users and groups services on all tests that need them #7620
- Do not catch table or registry exceptions when running tests #7621
- Remove unnecessary string copy #7625
- Fix system-info support for Unicode characters on Windows #7626
- libs: Update sqlite to version 3.38.5 #7628
- libs: Update OpenSSL to version 1.1.1o #7629
wmi_bios_info: Include Win32_BIOS attributes for all systems #7631- Port
memory_devicestable to Windows #7633 - Improve config parsing and osqueryfuzz-config performance #7635
- Implement a split and trim function using std::string_view #7636
- deb_packages: Do not display arch info in the package name #7638
- Fix thrift server shutting down when dropping privileges #7639
- Update
shared_resourcestable to add type names, fix type/maximum_allowed handling #7645 - Fix AWS certificate verification failing on all services #7652
- time: Fix the Windows local_timezone column value #7656
- Port platform_info table to M1 Macs #7660
- Remove the lldp_neighbors table #7664
- ci: Update osquery-packaging commit to the latest one #7667
- cmake: Add an option to enable or disable using ccache #7671
- cmake: Prevent defining some Linux only targets on other platforms #7672
- libs: Update OpenSSL to version 1.1.1q #7674
- Add documentation about 3rd-party dependency security #7684
- tpm_info: Refactor, ensure boolean values are always up to date #7686
- Port the
secureboottable to macOS #7692 - Fix a crash when parsing ATC config with no columns #7693
- ci: Update and temporarily disable the macOS Catalina test job #7700
- test: Fix Mdfind.test_sanity flakyness #7701
- Fix bug in GetHomeDirectories filesystem function #7705
- Update minimum macOS support from 10.12 to 10.14 #7707
- Add
firmware_typecolumn toplatform_infotable on Windows. #7710 - Fix
GetMemorySizefor Windowsmemory_devicestable #7711 - Correct macOS version support check for unified_log.mm #7713
- Improvements to osquery AWS logic #7714
- Temporarily disable memory_devices integration test #7717
- Add validation integration test for memory_devices #7722
- Increase mdfind query timeout to 30 seconds #7725
- platform_info: Add
firmware_typeto macOS #7727 - libs: Update libxml2 to v2.9.14 #7729
- libs: Update sqlite to version 3.39.2 #7736
- mdfind: Reduce table overhead and support quick interruption #7738
- test: Fix platform-info.test_sanity on Windows #7742
secureboot: Acquire the necessary process privileges on Windows #7743- ci: Migrate jobs from ubuntu-18.04 to ubuntu-20.04 #7745
- Fix a leak and improve users and groups APIs on Windows #7755
- Fix
process_file_eventssubscriber being incorrectly initialized #7759 - docs: Correct the description on how to configure and use Yara signature urls #7769
- build: Remove unused find_packages modules and submodule #7771
- misc: Delete temporary CTest files #7782
- ci: Add a job and helper scripts to periodically scan for CVEs #7787
processes: Stabilize thestart_timecolumn value on macOS and Linux #7788- ci: Update how we set github workflow step outputs #7791
- Fix deadlock when logging happens during a database reset #7798
- Fix handling of some errors during an AWS HTTP request #7811
- ci: Fix python version when installing modules and testing on macos #7813
- processes: Fix the procfs memory unit kB, which is 1024 bytes not 1000 #7818
- Do not access the AWS SDK request content type if missing #7834
- ci: Update some actions to remove deprecation warnings #7864
- cve: Ignore zstd CVE-2021-24031 #7865
- docs: Update the list of pages #7866
- libs: update Thrift to 0.17 #7868
- cve: Ignore libcryptsetup cves #7871
- cve: Ignore libdpkg CVE-2022-1664 #7872
- cve: Ignore libgcrypt cves #7873
- libs: Update zlib to 1.2.13 #7874
- libs: Update libarchive to 3.6.2 #7877
- Docs: mention the recent adoption of automatic CVE scanning #7878
- cmake: Remove forced static libraries search for osquery-toolchain #7881
- libs: Update libxml2 to 2.10.3 #7882
- git: Ignore compile_commands.json and pyrightconfig.json #7885
- ci: Automatically cancel old PR jobs #7887
- test: Fix flaky test_daemon_sigint #7888
- test: Add an option to run only selected python testcases #7890
- CHANGELOG 5.7.0 #7894
- ci: Improve error message when a library is missing from the manifest #7899
- pallets/werkzeug
- pypi/warehouse
- python-version: bump to 3.8.9 #10626
- docs/getting-started: formatting fixes, add macOS troubleshooting #10627
- Interfaces and services for JWK management #10628
- docs/application: update the project structure #10634
- Models, routes and views for creating OIDC publishers #10753
- Add
ExpiryCaveat#11122 - warehouse, tests: pick DB changes from #11122 #11157
- Refactor: Migrate to 2.0-style security policies #11218
- OIDC: More claims for GitHub’s provider #11239
- GitHub OIDC: validate
job_workflow_ref#11263 - OIDC macaroon minting for GitHub #11272
- Revert #11313 #11315
- macaroons/security_policy: avoid exceptions during user lookup #11322
- tests, warehouse: avoid potential recursion in authenticated_userid #11333
- warehouse, tests: check that session’s user still exists #11341
- warehouse, tests: require a matched route for session auth #11351
- Add a caveat for project IDs #11857
- API: Add “summary” field to vulnerability reports #11858
- Better upload errors when using API tokens #11885
- warehouse, tests: remove the journal view #11962
- vulnerabilities: expose withdrawn state on vulnerabilities #12443
- warehouse: add initial pending OIDC provider models #12572
- python/cpython
- pypa/pip
- rust-lang/rust
- rust-lang/rustc-dev-guide
- rust-lang/rustfix
- sarugaku/resolvelib
- snipe/snipe-it
- vehemont/nvdlib
- vityafx/serde-aux
- zulip/zulip
Software testing tools
- AFLplusplus/AFLplusplus
- AFLplusplus/LibAFL
- assert-rs/trycmd
- cgaebel/pipe
- GaloisInc/FAW
- garyttierney/intellij-ghidra
- google/gofuzz
- google/oss-fuzz
- googleprojectzero/weggli
- gstrauss/mcdb
- Homegear/Homegear
- jkrh/kvms
- kometchtech/docker-build
- lief-project/LIEF
- Manishearth/compiletest-rs
- microsoft/binskim
- ned14/quickcpplib
- NationalSecurityAgency/ghidra
- pwndbg/pwndbg
- update unicorn to 2.0.0 #1034
- colorful tip of the day #1046
- Fix aarch64 regs display #1054
- Fix context args crash on missing instruction #1055
- Remove shell commands registration #1064
- Improve search –next speed and add –trunc-out flag #1066
- Revert “Remove shell commands registration” #1073
- small refactor of vmmap module #1078
- Fix coredump debugging #1079
- Revert “Refactor heap code” #1084
- fix vis_heap_chunk test on CI? #1086
- Fix heap test binaries build #1087
- Remove QuietSloppyParsedCommand once and for all #1091
- tests.sh: add [filter] and –pdb #1092
- black all da code #1103
- fix #1098: dX cmds trunc out on x86 binaries #1104
- vmmap: use pwndbg.info.auxv instead of gdb.execute #1107
- ArgparsedCommand: fix
help cmdandcmd –helpbehavior #1108 - improve start and entry commands description #1109
- fix errno command #1112
- fix #1111 errno command edge case #1126
- fix distance command #1146
- fix qemu vmmap showing coredump mappings #1148
- Improve vmmap on coredump files #1149
- add patch command #1150
- Fix #1153 nextproginstr command #1158
- Show arch and emulation status on disasm banner #1160
- Fix #1165: set context-clear-screen on resetting scrollback #1166
- silence heap_bugs.c build warnings #1172
- Enhance heap with for static-linked binaries & remove typeinfo bloat #1173
- search command: remove unused string optional arg #1180
- Fix disable_colors formatting & test ctx disasm showing fds #1186
- fix #1190: telescope -r with addr as count #1198
- tips: add set show-flags on tip #1200
- add show-flags and show-compact-regs to ctx regs banner #1201
- remove defcon.py #1203
- bugreport command: use code listings #1204
- Bump gdb pt dump #1205
- Delete .sublime-settings #1206
- Update README with GDB build steps #1220
- fix #1221: ipi command multi-line inputs #1222
- events.py: remove unused Pause class #1223
- Fix #1197: dont display ctx on reg/mem changes #1239
- allow setting gdblib.regs.= #1267
- Fix #1256: fixes next cmds hangs on segfaults #1268
- Fix #1189: fixes patch command’s arch=… value #1269
- Pwndbg configuration: do not set history expansion #1292
- Fix parameter default values #1307
- Fix invalid zig path in tests makefile & suppress compilation warning #1308
- Increase CI timeout to 20 minutes #1309
- Fix setting empty ctx sections #1310
- lint.sh: lint only pwndbg files #1312
- fix lint #1356
- tests.sh: del joblog if –keep not passed #1360
- Fix lexer for coloring negative numbers in asm #1367
- Merged #1351 PR: Run tests in docker images #1370
- Remove instr operands padding in enhance #1372
- tips.py: add tip about Pwndbg’s signal handling #1373
- Fix tests reporting in parallel execution #1379
- tests zig cc: silence unused vars warnings #1382
- Fix debian10 ci #1383
- fix test_loads_binary_with_core_without_crashing on debian10 #1389
- tests reference-binary.c: dont rely on connect to 1.1.1.1 #1390
- Fix vmmap coredump test #1391
- version.py: fix build_id after recent refactors #1393
- fix #1188: incorrect 32-bit syscall display on x64 #1407
- abi.py: don’t recreate ABI dicts #1408
- Fix #1399: cymbol command on old GDB #1409
- tests.sh: fix –pdb (set SERIAL when –pdb is set) #1410
- Fix archlinux ci tests #1411
- returntocorp/semgrep-rules
- rust-fuzz/book
- rust-lang/rust-clippy
- Change
unnecessary_to_ownedinto_itersuggestions toMaybeIncorrect#8201 - Format
if_chaininvocations in clippy_utils #8370 - Fix some
unnecessary_filter_mapfalse positives #8479 - Add
unnecessary_find_maplint #8489 - Fix
unncessary_to_ownedfalse positive #8509 - Add
crate_in_macro_deflint #8576 - Extend
extra_unused_lifetimesto handle impl lifetimes #8737 - Address
unnecessary_to_ownedfalse positive #8794 - Optionally allow
expectandunwrapin tests #8802 - Improve “unknown field” error messages #8823
- Check
.fixedpaths’ existence inrun_ui#8844 - Add test for #8855 #8857
- Fix
empty_line_after_outer_attributefalse positive #8892 - Fix
extra_unused_lifetimesfalse positive #9037 - Enhance
needless_borrowto consider trait implementations #9136 - Add
ui_cargo_toml_metadatatest #9216 - Fix
to_string_in_format_argsfalse positive #9259 - Further enhance
needless_borrow, mildly refactorredundant_clone#9386 - Upgrade
compiletest-rsdependency #9523 - Expand internal lint
unnecessary_def_path#9566 - Fix bug introduced by #9386 #9635
- Fix
needless_borrowfalse positive #9674 - Add
lintcheckto packages linted bydogfoodtest #9691 - Improve
possible_borrower#9701 - Fix
needless_borrowfalse positive #9710 #9711 - Improve
needless_lifetimes#9743 - Update CONTRIBUTING.md with changelog guidance #9753
- Address issues 9739 and 9782 #9791
- Fix #9771 (
unnecessary_to_ownedfalse positive) #9796 - Fix typo in
expect_usedandunwrap_usedwarning messages #9863 - Move
line_spanto source.rs #9873 - Use
walk_generic_arg#9930 - Fix 10021 #10027
- Change
- S2E/s2e-env
- S2E/s2e
- unicorn-engine/unicorn
- Z3Prover/z3
Blockchain software
- coral-xyz/anchor
- ethereum/hevm
- Cleanup flake.nix, fix nix-build on macOS #34
- Cleanup GHC2021 standard extensions #42
- Code simplifications in EVM.SMT #43
- Performance: BA.unpack -> BA.convert #45
- Make Ethereum tests pass #56
- Move nixpkgs to unstable and clean up #72
- Change gas to Word64 to improve performance #73
- Remove sbv from cabal dependencies #88
- metaplex-foundation/metaplex
- OpenZeppelin/openzeppelin-contracts
- primitivefinance/rmm-core
- solana-labs/rbpf
- ton-blockchain/ton
