Background and aims
The Australian Capital Territory (ACT) Government sought the services of Global Research to review the already-completed analysis of comments received during a public engagement consultation. We were asked to review and assess the suitability of both the method and the coding process so that the client could be sure that they had received robust and trustworthy work.
Our role
Global Research agreed to peer-review the coding and to report on its relevance and effectiveness as well as to note any recommendations for the client. We were tasked with designing a method to test coding and themes, and to ascertain the effectiveness of these. Global Research, as an experienced practitioner of qualitative analysis, was approached to carry out this important quality-check assessment.
Design and method
To assess the coding, we first blind coded (that is, carried out coding without first checking how the other coding had been performed) and then compared the results to the original. To do this we removed the coding from the coding existing framework and test-coded a range of randomly selected comments from each of the data sources (survey, comment boards, submissions, and workshop data). These were then checked against each other, and wherever variance was seen, this was outlined and explained where possible.
Result
The results returned an acceptable (minimal) variance level, and Global Research concluded that the original coding criteria was applied appropriately and consistently. Any variances were explained by the depth of understanding that respective coding teams had with the project data and the differences in interpretation that this resulted in.
Outcome
The client received a detailed comparison assessment document that showed the coding variance for each batch of data. We were able to confirm that their processes are effective and that their coding criteria were well defined and applied consistently. The quality control check that the ACT Government proactively applied to this work demonstrates their commitment to excellence and transparency.