Labour productivity assessment is the key to optimal planning and allocation of resources for a construction project. In this study, a detailed questionnaire covering major factors influencing work productivity has been identified by surveying the labourers and their supervisors. The relative importance index (RII) has been used to consolidate the responses and identify the key factors. Based on the factors identified a field study is conducted to develop an automatic productivity assessment tool. The worker performance is estimated using data collected from wearable devices. The data is calibrated against the site supervisor estimated labour performance by training a neuro-fuzzy interference system. The developed model can serve as a framework for the automatic assessment of labour performance in a construction site. This can find extensive applications in making the resource allocation process in major construction sites more rational.