Micro Ultrasonic Machining (USM) is a promising micromachining process to generate micro features in brittle and hard materials. However, the factors determining the performance of micro USM are not fully characterized. In this study, the influence of workpiece materials on machining performance of micro USM is investigated. Representative hard and brittle materials (silicon, glass, hot pressed silicon carbide and sintered alumina) are selected as workpiece materials and tungsten as tool material. It is found that the material removal rate generally increases with the decrease of the material fracture toughness. Tool wear rate is insignificantly affected by the type of workpiece material. The microstructure and localized fracture mechanism of materials seem to affect the machined surface characteristics. Low fracture toughness of workpiece material leads to easy edge cracking of machined feature. The strength of intergranular bonding in ceramics may affect the machining mode, and, therefore, material removal rate and generated surface.