where ( \alpha ) is a learned or fixed extrapolation coefficient (set to 0.5 in our experiments). This linear correction term approximates the gradient at the cloud's version without recomputing forward pass. Edge and cloud maintain version counters ( v_e, v_c ). The cloud applies updates immediately. The edge applies received deltas in order but without locking. To prevent divergence, we use a soft reconciliation step every ( R ) iterations:
where ( \sigma^2 ) is gradient noise variance. This matches the rate of synchronous SGD when ( \tau ) is bounded.
[ T_\textseq = T_\textsend + T_\textforward + T_\textbackward + T_\textrecv ]
of SimulTrain is that the forward pass of one batch and the backward pass of a previous batch can overlap in time, if we carefully manage parameter versions and gradients. This is analogous to CPU pipelining but applied to distributed training across heterogeneous compute nodes. simultrain solution
SimulTrain matches centralized accuracy within 0.5%, while FedAvg drops by ~3% due to local overfitting. Removing gradient forecast causes divergence after 500 steps (accuracy falls to 45%). Removing weight reconciliation increases staleness indefinitely, leading to 12% higher loss. 7. Discussion Why does SimulTrain work? The key is the forecast+reconciliation loop. Forecast reduces bias, reconciliation prevents catastrophic staleness. The pipeline ensures that both edge and cloud are always busy, achieving near-optimal utilization.
In edge-cloud setting, data is at edge, compute is in cloud. The sequential round-trip time is:
[ \tilde\nabla_k = \nabla \ell(w^(e)_k; x_k) + \alpha \cdot (w^(c)_k - w^(e)_k) ] where ( \alpha ) is a learned or
[ w_t+1 = w_t - \eta \nabla \ell(w_t; x_t, y_t) ]
where ( T_\textsend ) and ( T_\textrecv ) depend on bandwidth, and ( T_\textforward, T_\textbackward ) on model size. For large models (e.g., ResNet-50), ( T_\textsend \gg T_\textforward ) on typical 4G/5G networks.
[ \mathbbE[|\nabla \ell(w^(c)_K)|^2] \leq \frac2L(f(w^(c)_0) - f^*)K\eta + O(\eta \sigma^2) + O(\tau^2 \eta^2) ] The cloud applies updates immediately
SimulTrain sends activations (lower dimension than raw data but higher than gradients). However, it enables bidirectional overlap , reducing total bandwidth-time product by 65% compared to SyncSGD. | Dataset | Centralized | SyncSGD | FedAvg (5 local steps) | SimulTrain | |-------------|-------------|---------|------------------------|------------| | UCF-101 | 84.2% | 83.9% | 81.1% | 83.7% | | WISDM | 91.5% | 91.3% | 88.9% | 91.1% |
SimulTrain reduces latency by 78% on 4G and 71% on 5G compared to SyncSGD. FedAvg hides latency via local steps but suffers from model drift. | Method | Upload per step (KB) | Download per step (KB) | |----------------|----------------------|------------------------| | Centralized | 7,500 (video frame) | 75 (weights) | | SyncSGD | 75 (gradients) | 75 (weights) | | SimulTrain | 30 (activations) | 75 (delta weights) |
[ w^(e) \leftarrow \beta w^(e) + (1-\beta) w^(c) ]
Proof sketch: The forecast term cancels first-order bias from staleness. Weight reconciliation prevents error accumulation. The pipeline yields the same effective gradient steps per unit time. Hardware: Edge = Raspberry Pi 4 (4GB RAM), Cloud = AWS g4dn.xlarge (NVIDIA T4). Network: emulated 4G (50 Mbps, 30 ms RTT) and 5G (300 Mbps, 10 ms RTT).
%!s(int=2026) © %!d(string=True Thread)
