Silver demonstrates the unique optical properties that make it suitable for various plasmonic applications. Determining the accurate optical constants for silver and investigating its dependence on grain structures are necessary for reliable theoretical predictions. This work investigates the effects of different deposition conditions and the resultant variable grain sizes, upon the optical properties of the silver films in conjunction with ellipsometry and optical characterization. Annealing conditions of substrate SiN x films were also studied to ensure good quality of deposited polycrystalline silver films. The effects of grain sizes of silver on its electronic relaxation times were studied and shown to be consistent with the theoretical electrical resistivity model. This study shows that voids and surface layers can affect the optical constant values and that reproducible results can be obtained by avoiding such artefacts. The results will have implications on the analysis of different silver based nanostructures like nanowires and nanorods where grain boundary scattering can affect the optical and electrical properties like reflection and resistivity.