Passive habitat restoration that depends on the natural dispersal of a target species to project sites is reliant on two major sets of processes: those that influence the quantity and timing of the arrival of propagules and those that affect survival and growth post recruitment. Particularly relevant for passive restoration projects is an understanding of the relative importance of rate of propagule arrival vs. post-settlement survival, as the same location might not be ideal for both sets of processes, however these data are often unavailable. Restoration projects are often not monitored more than one year post-construction, with the result that predictors of longer-term success may not be clear. We monitored native oysters (Ostrea lurida) that naturally recruited to restoration substrates for five years at two sites in San Francisco Bay. We report on changing rates of recruitment and adult densities over time that show different trajectories at different locations. We document gradients in oyster predation, competition, and physical stress along a short elevation span that correspond with differences in adult densities. These results demonstrate that for restoration to be successful given these gradients as well as interannual and site-specific variability, it is critical to take a portfolio approach that involves using a range of sites and tidal elevations to ensure success at the end of the project. Additionally, our longer-term dataset demonstrates the differences in conclusions we might have drawn from short-term monitoring (<2 years) compared with the understanding gained after 5 years.