We de novo assembled 12 G. barbadense accessions that span the wild-to-domesticated continuum, and constructed a graph-based pangenome. Whole genome comparative analyses identified 350,995 non-redundant structural variations (SVs) by integrating 16 G. barbadense and 12 other tetraploid cotton accessions. We clarified the genetic relationships within G. barbadense and identified selection signals using 672 G. barbadense genomes for which re-sequencing data were available. Interspecies introgression originated from Caribbean Sea region and subsequently selection drove the genomic differentiation on chromosome A01, which provided insight for understanding improvement in G. barbadense. Furthermore, SV-based genome-wide association study (GWAS) analysis identified four, three, and seven causal structural variations for fiber length, fiber strength and lint percentage, respectively. Two pleiotropic favorable alleles were combined into Xinhai cotton (Xinjiang cultivar) to significantly improve fiber length and fiber strength, and three favorable alleles were combined into Pima cotton, which significantly improved lint percentage.