Fatigue cracking and failure of inelastic heterogeneous asphalt concrete mixtures were modeled computationally with the finite element method. The model incorporates elastic behavior of the aggregate particles, viscoelastic behavior of the asphalt matrix, and time-dependent fracture both within the asphalt matrix and along boundaries between matrix and aggregate particles. Rate-dependent progressive cracking up to failure was implemented by incorporation of a cohesive zone fracture model. The resulting model was used to simulate comprehensive fatigue damage-associated mechanical behavior including microcracking, macrocracking, and eventual sample failure of several asphalt mixtures composed of different mixture constituents, which results in different damage evolution characteristics. Simulation results were compared with real fatigue testing data in both load-controlled and displacement-controlled modes and demonstrated good correlations to laboratory data with model calibrations. The approach proposed can be employed to predict complex fatigue behavior of asphalt mixtures by measurement of only fundamental material properties and fracture or damage properties of mixture constituents without recourse to expensive laboratory fatigue tests.