The exact nature of the heresy is debated, and can only be guessed at by the content of Paul's counter arguments in his letter. There are two broad views on the nature of the heresy. One is that it was a Jewish heresy, with a mystical and ascetic form of Judaism. The other was that it was a syncretic gnostic teaching. A few scholars however debate the existence of an actual heresy.
A Jewish heresy
In this view, the heresy is thought to reflect a teaching with a focus on Jewish covenant distinctiveness and privilege. Verses such as 2:11, 2:16 and 2:21 suggest a Jewish legalism. Paul reminds the church in 3:11 that where there can't be Greek and Jew, circumcision and uncircumcision, barbarian, Scythian, bondservant, freeman; but Christ is all, and in all.
A gnosticizing syncretism heresy
In this view, the heresy is thought to reflect a teaching on a special wisdom and knowledge from cosmic powers (Colossians 2:10). It may have been that the Colossians were integrating a mystery religion with Christ. Verses that point away from the kind of exclusivism that exists in Gnostic thinking support this view, such as 1:23: if it is so that you continue in the faith, grounded and steadfast, and not moved away from the hope of the Good News which you heard, which is being proclaimed in all creation under heaven; of which I, Paul, was made a servant.. Other verses teaching against an elitism include 1:28 and 3:11.
Paul argued against these kinds of false teaching by showing that they diminished Jesus Christ. Using the language of the heresy (words such as sophia meaning wisdom, and gnosis meaning knowledge) he reminded the Colossians that Christ is all in all (Colossians 3:11).