Lwip mainstream

[fjtrujy]

Summary

Switch the lwIP dependency from the long-standing ps2dev/lwip fork
(branch ps2-v2.0.3) to upstream lwip-tcpip/lwip STABLE-2_0_3_RELEASE,
and absorb the small set of accommodations into ps2sdk so the lwIP source tree
stays unmodified going forward.

This is the foundation for moving to newer lwIP releases — it removes the fork
as a blocker. A follow-up stacked PR upgrades to lwIP 2.2.1 on top of this.

Why

The ps2dev fork carried 5 PS2-specific patches against lwIP 2.0.3 source. Most
of them masked workarounds we can express through lwipopts.h/glue code, and
some are no longer needed at all in current code paths. Owning patches against
upstream lwIP makes every future version bump painful (rebases, conflict
resolution against a moving target). Switching to clean upstream removes that
maintenance tax.

What changes

Switch to upstream lwIP 2.0.3:

• download_dependencies.sh: clone lwip-tcpip/lwip at STABLE-2_0_3_RELEASE
  instead of ps2dev/lwip at ps2-v2.0.3.

ps2sdk-side accommodations replacing the fork's patches:

• iop/tcpip/tcpip-base/include/arch/cc.h: pre-include <sys/time.h>.
  Upstream sockets.h references struct timeval without including it when
  LWIP_TIMEVAL_PRIVATE=0.
• iop/tcpip/tcpip-base/include/stdlib.h: minimal IOP shim that defines
  atoi via sysclib's strtol. Upstream lwIP expects atoi to be declared.
• iop/tcpip/tcpip-base/include/lwipopts.h: set
  LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT=1 so mem_free() uses
  SYS_ARCH_PROTECT rather than a mutex. Required because SMAP RX calls
  pbuf_free() from interrupt-disabled context — same effect as the fork's
  mem.c patch but achieved purely via lwipopts.

Driver-side fixes uncovered by the move:

• iop/network/smap: handle 2-byte-aligned PBUF_RAM payloads in
  CopyToFIFO / CopyFromFIFO using lwl/lwr (R3000A has no unaligned
  word access — pbufs landing on a 2-byte boundary tripped AdEL on real
  hardware).
• iop/network/smap/src/main.c: prune dead PRE_LWIP_130_COMPAT branches.
• iop/tcpip/tcpip-netman: drop dead PRE_LWIP_130_COMPAT, advertise
  NETIF_FLAG_ETHERNET on the netif.

Configuration tightening for the IOP target:

• Disable IP_REASSEMBLY and IP_FRAG: PS2 networking targets a local LAN
  with MTU=1500 and TCP negotiates MSS=1460, so fragmentation never arises in
  the common case. Frees MEMP_NUM_REASSDATA / MEMP_NUM_FRAG_PBUF pool
  entries.
• Prune lwipopts.h entries that just restate lwIP 2.0.3 defaults
  (separate cleanups for the IOP and EE variants).
• Style: one-object-per-line formatting in ps2api/ps2ip Makefile lists.
• Move sys/time.h include from the Makefile's -include to arch/cc.h so
  it's visible to every consumer of arch/cc.h, not just lwIP source.

Compatibility

• The build still produces ps2ip-nm.irx and ps2ip.irx with the same export
  table; downstream IRXs (smap, netman) rebuild without changes to their
  imports.
• ps2link and other consumers continue to work without modification.

Test plan

• Build the full IOP networking chain: tcpip-netman → smap → netman,
  install IRXs, build ps2link.
• Real hardware: confirmed working — ps2link execee and reset succeed
  the same as with the fork's 2.0.3.

Follow-up

A stacked PR upgrades on top of this branch to lwIP 2.2.1, adding only what's
required (the load-bearing piece is a one-character fix in
common/include/errno.h — the section("data") attribute was creating a
section the IRX loader does not allocate, latent for years until upstream
removed LWIP_SOCKET_SET_ERRNO).

[sp193]

Is it really a good idea, to work around unaligned memory accesses in SMAP? I recall that LWIP had some options to deal with alignment limitations of some hosts, but it has been a very long time since I touched this, indeed.

Copying with the IOP is very slow.

[sp193]

Re: LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT

I may be wrong if things have changed, but we are not always using a mutex because semaphores are very slow on the IOP, compared to just disabling interrupts. Signalling a semaphore from an interrupt context is possible with iSignalSema().

[fjtrujy]

Re: LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT

I may be wrong if things have changed, but we are not always using a mutex because semaphores are very slow on the IOP, compared to just disabling interrupts. Signalling a semaphore from an interrupt context is possible with iSignalSema().

Good question.... I didn't think about that.
I was just looking to use mainstream changes to make future upgrades easier....
Anyway, I didn't notice any slowdown, but it is true that the only test that I did was using ps2link & a transfer_speed test

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/time.h>
#include <fcntl.h>
#include <stdint.h>
#include <errno.h>
#include <string.h>

#define BUF_SIZE 32768

void SleepThread(void);

static double time_diff(const struct timeval *start, const struct timeval *end) {
    return (end->tv_sec - start->tv_sec) + (end->tv_usec - start->tv_usec) / 1e6;
}

static void run_read_mode(const char *path) {
    int fd = open(path, O_RDONLY);
    if (fd < 0) {
        fprintf(stderr, "open %s: %s\n", path, strerror(errno));
        return;
    }

    char *buf = malloc(BUF_SIZE);
    if (!buf) {
        fprintf(stderr, "malloc failed\n");
        close(fd);
        return;
    }

    ssize_t r;
    uint64_t total = 0;
    struct timeval start, now, last_print;
    gettimeofday(&start, NULL);
    last_print = start;
    uint64_t interval_bytes = 0;

    printf("-- READ mode (read syscall) --\n");
    while ((r = read(fd, buf, BUF_SIZE)) > 0) {
        total += (uint64_t)r;
        interval_bytes += (uint64_t)r;
        gettimeofday(&now, NULL);
        double since_last = time_diff(&last_print, &now);
        if (since_last >= 1.0) {
            double instant_mib = (interval_bytes / (1024.0 * 1024.0)) / since_last;
            double elapsed = time_diff(&start, &now);
            double avg_mib = (total / (1024.0 * 1024.0)) / (elapsed > 0 ? elapsed : 1e-9);
            printf("Elapsed: %.2fs  Instant: %.3f MiB/s  Avg: %.3f MiB/s  Read: %lu bytes\n",
                   elapsed, instant_mib, avg_mib, (unsigned long)total);
            fflush(stdout);
            last_print = now;
            interval_bytes = 0;
        }
    }

    gettimeofday(&now, NULL);
    double total_secs = time_diff(&start, &now);
    if (r < 0) {
        fprintf(stderr, "read error: %s\n", strerror(errno));
    }
    printf("READ Done. Read %lu bytes in %.6fs (%.3f MiB/s)\n\n",
           (unsigned long)total, total_secs, (total / (1024.0 * 1024.0)) / (total_secs > 0 ? total_secs : 1e-9));

    free(buf);
    close(fd);
}

static void run_fread_mode(const char *path) {
    FILE *f = fopen(path, "rb");
    if (!f) {
        fprintf(stderr, "fopen %s failed: %s\n", path, strerror(errno));
        return;
    }

    char *buf = malloc(BUF_SIZE);
    if (!buf) {
        fprintf(stderr, "malloc failed\n");
        fclose(f);
        return;
    }

    size_t r;
    uint64_t total = 0;
    struct timeval start, now, last_print;
    gettimeofday(&start, NULL);
    last_print = start;
    uint64_t interval_bytes = 0;

    printf("-- FREAD mode (stdio fread) --\n");
    while ((r = fread(buf, 1, BUF_SIZE, f)) > 0) {
        total += (uint64_t)r;
        interval_bytes += (uint64_t)r;
        gettimeofday(&now, NULL);
        double since_last = time_diff(&last_print, &now);
        if (since_last >= 1.0) {
            double instant_mib = (interval_bytes / (1024.0 * 1024.0)) / since_last;
            double elapsed = time_diff(&start, &now);
            double avg_mib = (total / (1024.0 * 1024.0)) / (elapsed > 0 ? elapsed : 1e-9);
            printf("Elapsed: %.2fs  Instant: %.3f MiB/s  Avg: %.3f MiB/s  Read: %lu bytes\n",
                   elapsed, instant_mib, avg_mib, (unsigned long)total);
            fflush(stdout);
            last_print = now;
            interval_bytes = 0;
        }
    }

    gettimeofday(&now, NULL);
    double total_secs = time_diff(&start, &now);
    if (ferror(f)) {
        fprintf(stderr, "fread error\n");
    }
    printf("FREAD Done. Read %lu bytes in %.6fs (%.3f MiB/s)\n\n",
           (unsigned long)total, total_secs, (total / (1024.0 * 1024.0)) / (total_secs > 0 ? total_secs : 1e-9));

    free(buf);
    fclose(f);
}

int main(int argc, char **argv) {
    const char *path = "testfile.bin";
    const char *mode = "both"; // read, fread, both
    if (argc > 1) path = argv[1];
    if (argc > 2) mode = argv[2];

    if (strcmp(mode, "read") == 0) {
        run_read_mode(path);
    } else if (strcmp(mode, "fread") == 0) {
        run_fread_mode(path);
    } else {
        run_read_mode(path);
        run_fread_mode(path);
    }

    SleepThread(); // Prevent exit to see final output on PS2
    return 0;
}

[sp193]

I was referring to the comment you made in the code. There are various good reasons to enable this setting.

If you revert enough settings (including turning off core locking), you should eventually end up with an LWIP version that performs incredibly poorly on the IOP.

Personally, I feel that we need to strike a balance between using the stock code and providing a custom version. This isn't a modern game console, so the code not fitting its architecture can be a very bad thing.

Re: the PBUF alignment problem that I commented on above. I wrote a comment about how it was fixed, a long time ago. Without this change, the memory buffer will be unaligned.

pbuf_alloc() will no longer align the payload field, for PBUF_RAM allocations.
	This is so that frames being output will not be unaligned.

	Note: this still doesn't solve all cases. For example, generating a UDP packet will result in an unaligned final address,
	as PBUF_TRANSPORT is a general "one size fits all" option.

Looking at that changelog, you should be careful because some changes came about due to incompatibilities with how the SCE IOP kernel works.

[uyjulian]

Is it really a good idea, to work around unaligned memory accesses in SMAP? I recall that LWIP had some options to deal with alignment limitations of some hosts, but it has been a very long time since I touched this, indeed.

Copying with the IOP is very slow.

It avoids some difficult-to-diagnose issues such as hanging. Speed can come later.
(I don't see a major difference in the speed in this PR, measured with iperf, anyways)

Re: LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT

I may be wrong if things have changed, but we are not always using a mutex because semaphores are very slow on the IOP, compared to just disabling interrupts. Signalling a semaphore from an interrupt context is possible with iSignalSema().

LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT enabled (as in this PR) does not use a mutex and instead disables/enables interrupts.

[fjtrujy]

Thanks for the careful review @sp193 — let me try to address both concerns, since I think we're actually aligned on what we want.

Re: LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT and the comment in code

@uyjulian is right — when this option is =1, lwIP's mem_free() uses SYS_ARCH_PROTECT(lev) for the critical section, not a mutex. And on our port, SYS_ARCH_PROTECT resolves to CpuSuspendIntr() — see iop/tcpip/tcpip-base/sys_arch.c:355:

sys_prot_t sys_arch_protect(void)
{
    sys_prot_t Flags;
    CpuSuspendIntr(&Flags);
    return Flags;
}

So enabling this flag is exactly the "disable interrupts instead of taking a semaphore" path you're describing.
With the flag=0 (the default), lwIP would wrap mem_free() in a sys_mutex_t, which on the IOP means a semaphore — that's the slow path we want to avoid.

If the in-code comment reads like we're introducing a mutex, I'm happy to reword it to make the direction explicit (e.g., "uses SYS_ARCH_PROTECT (interrupt disable on IOP) rather than wrapping the critical section in a sys_mutex_t/semaphore"). Let me know if that helps.

Re: handling unaligned buffers in CopyToFIFO/CopyFromFIFO

Your historical pbuf-alignment comment is actually the reason I needed this commit. The ps2dev fork has a patch to pbuf_alloc() that adjusts payload alignment so that PBUF_RAM allocations land in a way that keeps the on-wire frame aligned. By moving to stock lwIP 2.0.3, that patch is gone — so there is now a real path where the PIO copy hits a halfword-aligned payload and the IOP traps an AdEL exception (reproducible: ExcCode=4, BadVAddr ending in 2/6/A/E, on the PIO fallback when len<64).

I agree pure IOP copy is slow, but:

• The aligned fast path is unchanged, so the common case has zero added cost.
• GCC turns the byte-wise word assembly into lwl/lwr, so even the slow path is reasonably tight (no per-byte function call overhead).
• @uyjulian measured with iperf and didn't see a meaningful delta.

My goal in this PR is exactly to stop patching the lwIP source so future bumps stay easy.

Re: balance between stock and custom
Totally agree — that's the line I'm trying to walk here. The places where the IOP architecture genuinely conflicts with stock lwIP interrupt-context pbuf_free, alignment, no <sys/time.h> from sockets.h) are kept as accommodations outside the lwIP tree.

The settings I pruned in the lwipopts cleanup commits are only the ones that already match upstream defaults or configure dead code — I verified the installed ps2ip-nm.irx is byte-identical (sha unchanged) before/after each cleanup commit, so nothing performance-critical was inadvertently flipped.

Happy to revisit any specific commit if you'd like — and to reword the lwipopts comment to be less ambiguous.

[uyjulian]

Aging tested with TCP iperf for 12 hours, works fine