Plant anthers are composed of different specialized cell types with distinct roles in plant reproduction. High temperature (HT) stress causes male sterility, resulting in crop yield reduction. However, the spatial expression atlas and regulatory dynamics during anther development and in response to HT remain largely unknown. Here, the first single-cell transcriptome atlas and chromatin accessibility survey in cotton anther are established, depicting the specific expression and epigenetic landscape of each type of cell in anthers. The reconstruction of meiotic cells, tapetal cells, and middle layer cell developmental trajectories not only identifies novel expressed genes, but also elucidates the precise degradation period of middle layer and reveals a rapid function transition of tapetal cells during the tetrad stage. By applying HT, heterogeneity in HT response is shown among cells of anthers, with tapetal cells responsible for pollen wall synthesis are most sensitive to HT. Specifically, HT shuts down the chromatin accessibility of genes specifically expressed in the tapetal cells responsible for pollen wall synthesis, such as QUARTET 3 (QRT3) and CYTOCHROME P450 703A2 (CYP703A2), resulting in a silent expression of these genes, ultimately leading to abnormal pollen wall and male sterility. Collectively, this study provides substantial information on anthers and provides clues for heat-tolerant crop creation.