The Constrained Application Protocol (CoAP) is a lightweight web transfer protocol designed based on the REST architecture standardized by the Internet Engineering Task Force (IETF) to meet and accommodate the requirements of the constrained Internet of Things (IoT) environments. Managing congestion control in a resource-constrained lossy network with a high bit error rate is a significantly challenging task that needs to be addressed. The primary congestion control mechanism defined by CoAP specification leverages on basic binary exponential backoff and often fails to utilize the network dynamics to the best of its traffic conditions. As a result, CoCoA has been introduced for better IoT resource utilization. In addition, CoCoA retransmission timeout (RTO) for network dynamics is based on constant coefficient values. The resource-constrained nature of IoT networks poses new design challenges for congestion control mechanisms. In this paper, we propose a new particle swarm optimization (PSO)-based congestion control approach called psoCoCoA as a variation of CoCoA. The psoCoCoA applies random and optimal parameter-driven simulation to optimize default CoAP parameters and update the fitness and velocity positions to adapt to the traffic conditions. This process is performed for different traffic scenarios by varying the retransmission and max-age values by using the optimization-based algorithm. We carried out extensive simulations to validate the congestion control performance for CoAP with Observe, CoCoA, and psoCoCoA with different network topologies. The results indicate that psoCoCA outperforms or very similar to CoCoA and achieves better performance compared to CoAP with Observe under different network scenarios. © 2021 John Wiley & Sons, Ltd.