Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the following Registration Statements:

(1) Registration Statement (Form S-8 No. 333-61190) pertaining to the Delta Apparel, Inc. 2000 Stock Option Plan and Delta Apparel, Inc. Incentive Stock Award Plan

(2) Registration Statement (Form S-8 No. 333-172018) pertaining to the Delta Apparel, Inc. 2010 Stock Plan

of our reports dated September 28, 2012, with respect to the consolidated financial statements and schedule of Delta Apparel, Inc. and subsidiaries and the effectiveness of internal control over financial reporting of Delta Apparel, Inc. and subsidiaries, included in the Annual Report on Form 10-K for the year ended June 30, 2012.

Atlanta, Georgia

September 28, 2012